Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-5090

troubles on smp/multicore systems with mariadb+pam+ldap+openssl

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 5.5.33
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Description

      Version: '5.5.31-MariaDB' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Source distribution
      * glibc detected * /usr/sbin/mysqld: corrupted double-linked list: 0x00007f2088006a80 *
      = Backtrace:
      /lib64/libc.so.6(+0x793d6)[0x7f20ed3063d6]
      /lib64/libc.so.6(+0x79789)[0x7f20ed306789]
      /lib64/libc.so.6(+0x7a035)[0x7f20ed307035]
      /usr/lib64/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f2091d65254]
      /usr/lib64/libssl.so.1.0.0(SSL_CTX_free+0x180)[0x7f2092034928]
      /usr/lib64/libldap-2.4.so.2(ldap_int_tls_destroy+0x15)[0x7f20926ae936]
      /lib64/ld-linux-x86-64.so.2(+0x12966)[0x7f20eeabb966]
      /lib64/ld-linux-x86-64.so.2(+0x13423)[0x7f20eeabc423]
      /lib64/ld-linux-x86-64.so.2(+0xde3f)[0x7f20eeab6e3f]
      /lib64/libdl.so.2(+0x15cf)[0x7f20ede345cf]
      /lib64/libdl.so.2(dlclose+0x1f)[0x7f20ede3416f]
      /lib64/libpam.so.0(+0x59d6)[0x7f20932009d6]
      /lib64/libpam.so.0(pam_end+0x33)[0x7f20931fe503]
      /usr/lib64/mysql/plugin/auth_pam.so(+0xb6c)[0x7f2093408b6c]
      /usr/sbin/mysqld[0x54bfc5]
      /usr/sbin/mysqld(_Z16acl_authenticateP3THDjj+0x895)[0x55edd3]
      /usr/sbin/mysqld[0x634615]
      /usr/sbin/mysqld(_Z16login_connectionP3THD+0x38)[0x63562c]
      /usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x21)[0x635c20]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x136)[0x635fa8]
      /usr/sbin/mysqld(handle_one_connection+0x42)[0x636095]
      /lib64/libpthread.so.0(+0x8bbe)[0x7f20ee895bbe]
      /lib64/libc.so.6(clone+0x6d)[0x7f20ed36ec9d]
      

      details:

      /etc/mysql/my.cnf:

      ...
      [mysqld]
      plugin-load=auth_pam.so
      ...
      

      /etc/pam.d/mysql:

      auth required pam_ldap.so config=/etc/ldap_mysql.conf debug
      account required pam_ldap.so config=/etc/ldap_mysql.conf debug
      

      /etc/ldap_mysql.conf:

      uri ldaps:ldap.xxx ldaps:ldap2.xxx
      ssl on
      scope sub
      

      stress test

      :
      #!/bin/sh
      ...
      for((i=0;i<100;i++)); do
       for((y=0;y<10;y++)); do
        echo 'show processlist;'|mysql -u uuu --password=ppp -h hhh &
       done
      done
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              JWSmith John W Smith added a comment -

              I think we just encountered this (or a very similar issue) with 5.5.38 running on Ubuntu 12.04 LTS.

              Server version: 5.5.38-MariaDB-1~precise-log
              key_buffer_size=8388608
              read_buffer_size=131072
              max_used_connections=9
              max_threads=202
              thread_count=4
              It is possible that mysqld could use up to
              key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1692418 K bytes of memory
              Hope that's ok; if not, decrease some variables in the equation.

              Thread pointer: 0x0x7f1f9c424000
              Attempting backtrace. You can use the following information to find out
              where mysqld died. If you see no messages after this, something went
              terribly wrong...
              stack_bottom = 0x7f2853701e20 thread_stack 0x48000
              /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa7499b]
              /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6ccf88]
              /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f28564b2cb0]
              /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f2854c88425]
              /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f2854c8bb8b]
              /lib/x86_64-linux-gnu/libc.so.6(+0x7439e)[0x7f2854cc639e]
              /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f2854d5cf47]
              /lib/x86_64-linux-gnu/libc.so.6(+0x109e40)[0x7f2854d5be40]
              /lib/x86_64-linux-gnu/libc.so.6(+0x10aefe)[0x7f2854d5cefe]
              /lib/x86_64-linux-gnu/security/pam_ldap.so(+0x4587)[0x7f2575cfd587]
              /lib/x86_64-linux-gnu/security/pam_ldap.so(+0x1859)[0x7f2575cfa859]
              /lib/x86_64-linux-gnu/security/pam_ldap.so(pam_sm_authenticate+0x12c)[0x7f2575cfb48c]
              /lib/x86_64-linux-gnu/libpam.so.0(+0x2b45)[0x7f285480db45]
              /lib/x86_64-linux-gnu/libpam.so.0(pam_authenticate+0x38)[0x7f285480d3c8]
              /usr/lib/mysql/plugin/auth_pam.so(+0xcae)[0x7f1f881f6cae]
              /usr/sbin/mysqld[0x53f619]
              /usr/sbin/mysqld(_Z16acl_authenticateP3THDjj+0x9da)[0x556b1a]
              /usr/sbin/mysqld[0x651259]
              /usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x47)[0x652be7]
              /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x12a)[0x65316a]
              /usr/sbin/mysqld(handle_one_connection+0x4c)[0x6532bc]
              /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f28564aae9a]
              /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2854d463fd]

              Trying to get some variables.
              Some pointers may be invalid and cause the dump to abort.
              Query (0x0): is an invalid pointer
              Connection ID (thread ID): 6340
              Status: NOT_KILLED

              Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

              Show
              JWSmith John W Smith added a comment - I think we just encountered this (or a very similar issue) with 5.5.38 running on Ubuntu 12.04 LTS. Server version: 5.5.38-MariaDB-1~precise-log key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=9 max_threads=202 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1692418 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f1f9c424000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2853701e20 thread_stack 0x48000 /usr/sbin/mysqld(my_print_stacktrace+0x2b) [0xa7499b] /usr/sbin/mysqld(handle_fatal_signal+0x398) [0x6ccf88] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f28564b2cb0] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f2854c88425] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f2854c8bb8b] /lib/x86_64-linux-gnu/libc.so.6(+0x7439e) [0x7f2854cc639e] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37) [0x7f2854d5cf47] /lib/x86_64-linux-gnu/libc.so.6(+0x109e40) [0x7f2854d5be40] /lib/x86_64-linux-gnu/libc.so.6(+0x10aefe) [0x7f2854d5cefe] /lib/x86_64-linux-gnu/security/pam_ldap.so(+0x4587) [0x7f2575cfd587] /lib/x86_64-linux-gnu/security/pam_ldap.so(+0x1859) [0x7f2575cfa859] /lib/x86_64-linux-gnu/security/pam_ldap.so(pam_sm_authenticate+0x12c) [0x7f2575cfb48c] /lib/x86_64-linux-gnu/libpam.so.0(+0x2b45) [0x7f285480db45] /lib/x86_64-linux-gnu/libpam.so.0(pam_authenticate+0x38) [0x7f285480d3c8] /usr/lib/mysql/plugin/auth_pam.so(+0xcae) [0x7f1f881f6cae] /usr/sbin/mysqld [0x53f619] /usr/sbin/mysqld(_Z16acl_authenticateP3THDjj+0x9da) [0x556b1a] /usr/sbin/mysqld [0x651259] /usr/sbin/mysqld(_Z22thd_prepare_connectionP3THD+0x47) [0x652be7] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x12a) [0x65316a] /usr/sbin/mysqld(handle_one_connection+0x4c) [0x6532bc] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f28564aae9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f2854d463fd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): is an invalid pointer Connection ID (thread ID): 6340 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
              Hide
              dab1818 Dmitry Bakshaev added a comment -

              i found workaround for this problem:

              replace pam_ldap (http://www.padl.com/OSS/pam_ldap.html) with nss-pam-ldapd (http://arthurdejong.org/nss-pam-ldapd/)

              maybe this is not mariadb bug? pam_ldap related? close it?

              Show
              dab1818 Dmitry Bakshaev added a comment - i found workaround for this problem: replace pam_ldap ( http://www.padl.com/OSS/pam_ldap.html ) with nss-pam-ldapd ( http://arthurdejong.org/nss-pam-ldapd/ ) maybe this is not mariadb bug? pam_ldap related? close it?
              Hide
              elenst Elena Stepanova added a comment -

              John W Smith, did you also find any kind of a workaround? Does the one suggested by Dmitry Bakshaev work for you as well?

              Show
              elenst Elena Stepanova added a comment - John W Smith , did you also find any kind of a workaround? Does the one suggested by Dmitry Bakshaev work for you as well?
              Hide
              JWSmith John W Smith added a comment -

              We moved to MariaDB v10.0.x series, which does not exhibit the same issue. MariaDB v10.0.x series has worked flawlessly for several months in our development environment. We currently use the OS version of Centrify on our production MariaDB 5.5.x DB server. We plan on migrating production to Maria v10.0.x series soon, and plan to migrate those systems to pam_ldap at that time.

              Show
              JWSmith John W Smith added a comment - We moved to MariaDB v10.0.x series, which does not exhibit the same issue. MariaDB v10.0.x series has worked flawlessly for several months in our development environment. We currently use the OS version of Centrify on our production MariaDB 5.5.x DB server. We plan on migrating production to Maria v10.0.x series soon, and plan to migrate those systems to pam_ldap at that time.
              Hide
              dab1818 Dmitry Bakshaev added a comment -

              retested on mariadb-10.0.21 with pam_ldap-186.
              this bug reproduced in our environment.
              "dangerous mix" of openldap+openssl+threads?
              we move to nss-pam-ldapd...
              and nss-pam-ldapd has more other advantages for us.

              p.s.: also i need retest auth_pam with sssd (https://fedorahosted.org/sssd/)

              Show
              dab1818 Dmitry Bakshaev added a comment - retested on mariadb-10.0.21 with pam_ldap-186. this bug reproduced in our environment. "dangerous mix" of openldap+openssl+threads? we move to nss-pam-ldapd... and nss-pam-ldapd has more other advantages for us. p.s.: also i need retest auth_pam with sssd ( https://fedorahosted.org/sssd/ )

                People

                • Assignee:
                  elenst Elena Stepanova
                  Reporter:
                  dab1818 Dmitry Bakshaev
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated: