Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7838

Got error 5 "Input/output error" during COMMIT;

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Not a Bug
    • Affects Version/s: 10.0.17-galera
    • Fix Version/s: 10.0.19-galera
    • Component/s: Replication
    • Labels:
      None

      Description

      The environment is a mixed of Mysql5.6 and MariaDB 10.0. RBR replication from 5.6 to 10.0 is working fine except we have a large volume of data copy from one innodb table into another innodb table.

      example.

      INSERT into lgV3_Report.T_Report_Copy
      select * from lgV3_Report.T_Report where Date >='2015-2-20' AND Date < '2015-2-25';

      Which could be a few million of rows. All MySQL 5.6 salves are committed this query without problem. However, on a MariaDB 10.0.17 salve, the SQL_Slave thread stopped with this error

      "Got error 5 "Input/output error" during COMMIT;"

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            elenst Elena Stepanova added a comment - - edited

            I tried to reproduce with a simple table and 4M rows, it worked all right.

            Leo Kwan,
            Is it reproducible every time?
            If not, did you check for real input/output errors (maybe disk problems or something)?
            If yes and there was none, please

            • provide cnf files for master and slave;
            • specify the exact version of MySQL 5.6;
            • paste output of SHOW CREATE TABLE T_Report_Copy or whichever table you insert into;
            • paste output of SHOW TRIGGERS LIKE 'T_Report_Copy' or whichever table you insert into.
            • attach the error log from the slave which covers the time when the problem occurred (preferably the whole log from the server startup till shutdown or till now if it hasn't been shutdown since then).

            If any of this information is private, you can upload it to ftp.askmonty.org/private

            Thanks.

            Show
            elenst Elena Stepanova added a comment - - edited I tried to reproduce with a simple table and 4M rows, it worked all right. Leo Kwan , Is it reproducible every time? If not, did you check for real input/output errors (maybe disk problems or something)? If yes and there was none, please provide cnf files for master and slave; specify the exact version of MySQL 5.6; paste output of SHOW CREATE TABLE T_Report_Copy or whichever table you insert into; paste output of SHOW TRIGGERS LIKE 'T_Report_Copy' or whichever table you insert into. attach the error log from the slave which covers the time when the problem occurred (preferably the whole log from the server startup till shutdown or till now if it hasn't been shutdown since then). If any of this information is private, you can upload it to ftp.askmonty.org/private Thanks.
            Hide
            leomkkwan Leo Kwan added a comment -

            It can be reproducible every time.

            I got the exact error from the error log.

            150325 17:09:55 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073807360
            150325 17:09:56 [ERROR] WSREP: rbr write fail, data_len: 0, 2
            150325 17:11:16 [ERROR] Slave SQL: Got error 5 "Input/output error" during COMMIT, Internal MariaDB error code: 1180
            150325 17:11:16 [Warning] Slave: Got error 5 "Input/output error" during COMMIT Error_code: 1180
            150325 17:11:16 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'DB41-bin.000911' position 920434229
            

            This is a stand alone server running 10.0.17-MariaDB-wsrep-log. Running with the following config. wsrep is enabled, but it's only a one node cluster.

            wsrep_provider                  = /usr/lib64/galera/libgalera_smm.so
            wsrep_provider_options          = "gcache.size=100G; gcache.page_size=1G; gmcast.segment=1; gcs.fc_limit=256; gcs.fc_factor=0.99; gcs.fc_master_slave=yes;"
            wsrep_cluster_name              = "Dash"
            wsrep_cluster_address           = "gcomm://"
            

            I am not sure this is a bug or not now.

            Show
            leomkkwan Leo Kwan added a comment - It can be reproducible every time. I got the exact error from the error log. 150325 17:09:55 [Warning] WSREP: transaction size limit (1073741824) exceeded: 1073807360 150325 17:09:56 [ERROR] WSREP: rbr write fail, data_len: 0, 2 150325 17:11:16 [ERROR] Slave SQL: Got error 5 "Input/output error" during COMMIT, Internal MariaDB error code: 1180 150325 17:11:16 [Warning] Slave: Got error 5 "Input/output error" during COMMIT Error_code: 1180 150325 17:11:16 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START" . We stopped at log 'DB41-bin.000911' position 920434229 This is a stand alone server running 10.0.17-MariaDB-wsrep-log. Running with the following config. wsrep is enabled, but it's only a one node cluster. wsrep_provider = /usr/lib64/galera/libgalera_smm.so wsrep_provider_options = "gcache.size=100G; gcache.page_size=1G; gmcast.segment=1; gcs.fc_limit=256; gcs.fc_factor=0.99; gcs.fc_master_slave=yes;" wsrep_cluster_name = "Dash" wsrep_cluster_address = "gcomm: //" I am not sure this is a bug or not now.
            Hide
            elenst Elena Stepanova added a comment -

            Thanks. It's still important that you are using the wsrep version – from the log exceprt, it looks like the problem might have started there.

            Nirbhay Choubey,
            Could you please advise on this? Will the breach of transaction size limit end up with the 'input/output error', and if so, can it be avoided (e.g. is the limit configurable)?

            Show
            elenst Elena Stepanova added a comment - Thanks. It's still important that you are using the wsrep version – from the log exceprt, it looks like the problem might have started there. Nirbhay Choubey , Could you please advise on this? Will the breach of transaction size limit end up with the 'input/output error', and if so, can it be avoided (e.g. is the limit configurable)?
            Hide
            nirbhay_c Nirbhay Choubey added a comment -

            In galera cluster there is a limit set on the transaction size (1GB by default) that it can handle. It can be controlled by wsrep_max_ws_size system variable: https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/#wsrep_max_ws_size

            Show
            nirbhay_c Nirbhay Choubey added a comment - In galera cluster there is a limit set on the transaction size (1GB by default) that it can handle. It can be controlled by wsrep_max_ws_size system variable: https://mariadb.com/kb/en/mariadb/galera-cluster-system-variables/#wsrep_max_ws_size

              People

              • Assignee:
                nirbhay_c Nirbhay Choubey
                Reporter:
                leomkkwan Leo Kwan
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: