Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-2210

LP:1008293 - SET key_cache_segments leads to crashes in get_partitioned_key_cache_statistics, or safe_mutex_lock, or partitioned_key_cache_statistics

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      On 3 of the 4 servers i tried to do a "set global key_cache_segments = 8", 3 have crashed and only one didnt.

      The crashing servers were :

      • One on MariaDB 5.3.7 (immediate crash with "server has gone away" when sending the set command)
      • One on MariaDB 5.5.23 (set command returns "OK" but server crashes just after that)
      • One on MariaDB 5.5.24 (same as on the previous one)
        The server that has not crashed was on 5.5.23.

      All the servers are on Debian Squeeze amd64 using MariaDB repository packages.

      All servers have different datas and workloads (and are not using replication), these are used on a shared hosting environement so there is many databases/tables on each with many random accesses on them.

      Did put the value on my.cnf and restarted completely the servers and it didnt crash (some have been up for weeks without a crash), so it seems to be only crashing when changing the parameter on a live server.

      (attached two crash reports: 5.5.24 and 5.3.7)

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            jeanweisbuch Jean Weisbuch added a comment -

            Two crash reports from MariaDB error logs
            LPexportBug1008293_maria-key_cache_segment-crashreport.txt

            Show
            jeanweisbuch Jean Weisbuch added a comment - Two crash reports from MariaDB error logs LPexportBug1008293_maria-key_cache_segment-crashreport.txt
            Hide
            jeanweisbuch Jean Weisbuch added a comment -

            Re: Changing key_cache_segments value using set leads to crashes

            Show
            jeanweisbuch Jean Weisbuch added a comment - Re: Changing key_cache_segments value using set leads to crashes
            Hide
            elenst Elena Stepanova added a comment -

            Re: Changing key_cache_segments value using set leads to crashes
            Hi Jean,

            Could you please provide your my.cnf file from a server where you observed the crash?

            Thanks.

            Show
            elenst Elena Stepanova added a comment - Re: Changing key_cache_segments value using set leads to crashes Hi Jean, Could you please provide your my.cnf file from a server where you observed the crash? Thanks.
            Hide
            jeanweisbuch Jean Weisbuch added a comment -

            Re: Changing key_cache_segments value using set leads to crashes
            Here is one of the servers conf (the others have similar ones with mainly different buffer_pool_size, key_buffer_size, table_cache and max_connections) :

            [client]
            port = 3306
            socket = /var/run/mysqld/mysqld.sock
            [mysqld_safe]
            socket = /var/run/mysqld/mysqld.sock
            nice = 0
            [mysqld]
            user = mysql
            pid-file = /var/run/mysqld/mysqld.pid
            socket = /var/run/mysqld/mysqld.sock
            port = 3306
            basedir = /usr
            datadir = /var/lib/mysql
            tmpdir = /tmpfs
            skip-external-locking
            key_buffer = 512M
            max_allowed_packet = 64M
            thread_cache_size = 12
            myisam-recover = BACKUP
            max_connections = 150
            table_cache = 16384
            table_definition_cache = 16384
            query_cache_limit = 1M
            query_cache_size = 128M
            query_cache_strip_comments = 1
            local-infile = 0
            open_files_limit = 131072
            skip-name-resolve
            max_delayed_threads = 1
            wait_timeout = 60
            interactive_timeout = 60
            log_error = /var/log/mysql/error.log
            join_buffer_size = 1M
            read_buffer_size = 256K
            tmp_table_size = 32M
            max_heap_table_size = 32M
            max_connect_errors = 1000
            query_prealloc_size = 131072
            query_alloc_block_size = 16384
            myisam_sort_buffer_size = 4M
            concurrent_insert = 2
            read_rnd_buffer_size = 1M
            default_storage_engine = InnoDB
            innodb_buffer_pool_size = 1280M
            innodb_log_file_size = 32M
            innodb_flush_log_at_trx_commit = 2
            innodb_log_buffer_size = 4M
            innodb_file_per_table = 1
            innodb_open_files = 400
            innodb_file_format = barracuda
            key_cache_segments = 8 (wasnt there before i tried to set the value)
            log_slow_queries = /var/log/mysql/mysql-slow.log
            long_query_time = 2
            [mysqldump]
            quick
            quote-names
            max_allowed_packet = 16M
            [mysql]
            [isamchk]
            key_buffer = 16M
            !includedir /etc/mysql/conf.d/

            And in the conf.d dir :
            /etc/mysql/conf.d/mariadb.cnf:[client]
            /etc/mysql/conf.d/mariadb.cnf:[mysqld]
            /etc/mysql/conf.d/mysqld_safe_syslog.cnf:[mysqld_safe]
            /etc/mysql/conf.d/mysqld_safe_syslog.cnf:syslog

            Show
            jeanweisbuch Jean Weisbuch added a comment - Re: Changing key_cache_segments value using set leads to crashes Here is one of the servers conf (the others have similar ones with mainly different buffer_pool_size, key_buffer_size, table_cache and max_connections) : [client] port = 3306 socket = /var/run/mysqld/mysqld.sock [mysqld_safe] socket = /var/run/mysqld/mysqld.sock nice = 0 [mysqld] user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock port = 3306 basedir = /usr datadir = /var/lib/mysql tmpdir = /tmpfs skip-external-locking key_buffer = 512M max_allowed_packet = 64M thread_cache_size = 12 myisam-recover = BACKUP max_connections = 150 table_cache = 16384 table_definition_cache = 16384 query_cache_limit = 1M query_cache_size = 128M query_cache_strip_comments = 1 local-infile = 0 open_files_limit = 131072 skip-name-resolve max_delayed_threads = 1 wait_timeout = 60 interactive_timeout = 60 log_error = /var/log/mysql/error.log join_buffer_size = 1M read_buffer_size = 256K tmp_table_size = 32M max_heap_table_size = 32M max_connect_errors = 1000 query_prealloc_size = 131072 query_alloc_block_size = 16384 myisam_sort_buffer_size = 4M concurrent_insert = 2 read_rnd_buffer_size = 1M default_storage_engine = InnoDB innodb_buffer_pool_size = 1280M innodb_log_file_size = 32M innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 4M innodb_file_per_table = 1 innodb_open_files = 400 innodb_file_format = barracuda key_cache_segments = 8 (wasnt there before i tried to set the value) log_slow_queries = /var/log/mysql/mysql-slow.log long_query_time = 2 [mysqldump] quick quote-names max_allowed_packet = 16M [mysql] [isamchk] key_buffer = 16M !includedir /etc/mysql/conf.d/ And in the conf.d dir : /etc/mysql/conf.d/mariadb.cnf: [client] /etc/mysql/conf.d/mariadb.cnf: [mysqld] /etc/mysql/conf.d/mysqld_safe_syslog.cnf: [mysqld_safe] /etc/mysql/conf.d/mysqld_safe_syslog.cnf:syslog
            Hide
            sbester sbester added a comment -

            Very easy to reproduce. Any myisam workload with 2 or more threads will hit this. Try setting the key_cache_segments to a random value in 1 thread and select/insert/update/show global status/cache index in another thread.

            Tttached two examples of a crash.
            maria_bug_1008293_crashes.txt
            LPexportBug1008293_maria_bug_1008293_crashes.txt

            Show
            sbester sbester added a comment - Very easy to reproduce. Any myisam workload with 2 or more threads will hit this. Try setting the key_cache_segments to a random value in 1 thread and select/insert/update/show global status/cache index in another thread. Tttached two examples of a crash. maria_bug_1008293_crashes.txt LPexportBug1008293_maria_bug_1008293_crashes.txt
            Hide
            sbester sbester added a comment -

            Re: Changing key_cache_segments value using set leads to crashes
            Very easy to reproduce. Any myisam workload with 2 or more threads will hit this. Try setting the key_cache_segments to a random value in 1 thread and select/insert/update/show global status/cache index in another thread.

            Tttached two examples of a crash.

            Show
            sbester sbester added a comment - Re: Changing key_cache_segments value using set leads to crashes Very easy to reproduce. Any myisam workload with 2 or more threads will hit this. Try setting the key_cache_segments to a random value in 1 thread and select/insert/update/show global status/cache index in another thread. Tttached two examples of a crash.
            Hide
            elenst Elena Stepanova added a comment -

            Re: Changing key_cache_segments value using set leads to crashes
            Jean, Shane, thank you.

            Reproducible with the RQG grammar and command line below on maria/5.2 revno 3139, maria/5.3 revno 3539, maria/5.5 revno 3426, and on release versions of 5.2, 5.3, 5.5.

            1. cat 1.yy
              thread1:

            SET GLOBAL key_cache_segments = _digit;

            query:
            SHOW GLOBAL STATUS;

            1. End of RQG grammar
            1. Run as
              perl ./runall.pl \
              --grammar=1.yy \
              --duration=60 \
              --queries=10M \
              --threads=2 \
              --skip-gendata \
              --basedir=<your basedir> \
              --vardir=<your vardir>
            Show
            elenst Elena Stepanova added a comment - Re: Changing key_cache_segments value using set leads to crashes Jean, Shane, thank you. Reproducible with the RQG grammar and command line below on maria/5.2 revno 3139, maria/5.3 revno 3539, maria/5.5 revno 3426, and on release versions of 5.2, 5.3, 5.5. cat 1.yy thread1: SET GLOBAL key_cache_segments = _digit; query: SHOW GLOBAL STATUS; End of RQG grammar Run as perl ./runall.pl \ --grammar=1.yy \ --duration=60 \ --queries=10M \ --threads=2 \ --skip-gendata \ --basedir=<your basedir> \ --vardir=<your vardir>
            Hide
            elenst Elena Stepanova added a comment -

            Re: Changing key_cache_segments value using set leads to crashes
            The other variation of crash can be hit with the following grammar and command line:

            1. cat 2.yy
              query_init:
              SET GLOBAL keycache1.key_buffer_size = 1024*1024;
              thread1:
              SET GLOBAL key_cache_segments = _digit;

            query:
            CACHE INDEX _table IN keycache1;

            1. End of RQG grammar
            1. Run as:

            perl ./runall.pl \
            --grammar=2.yy \
            --duration=60 \
            --queries=10M \
            --engine=MyISAM \
            --threads=2 \
            --basedir=<your basedir> \
            --vardir=<your vardir>

            Show
            elenst Elena Stepanova added a comment - Re: Changing key_cache_segments value using set leads to crashes The other variation of crash can be hit with the following grammar and command line: cat 2.yy query_init: SET GLOBAL keycache1.key_buffer_size = 1024*1024; thread1: SET GLOBAL key_cache_segments = _digit; query: CACHE INDEX _table IN keycache1; End of RQG grammar Run as: perl ./runall.pl \ --grammar=2.yy \ --duration=60 \ --queries=10M \ --engine=MyISAM \ --threads=2 \ --basedir=<your basedir> \ --vardir=<your vardir>
            Hide
            ratzpo Rasmus Johansson added a comment -

            Launchpad bug id: 1008293

            Show
            ratzpo Rasmus Johansson added a comment - Launchpad bug id: 1008293

              People

              • Assignee:
                igor Igor Babaev
                Reporter:
                jeanweisbuch Jean Weisbuch
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: