Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-2582

LP:509521 - external (system) libevent cannot be used

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Build system always uses internal copy of libevent (we all know bundled libs are root of evil .

      Using always external is easy:
      http://cvs.pld-linux.org/cgi-bin/cvsweb/packages/mariadb/mariadb-libevent.patch
      but proper fix would be to allow to choose between internal and system one at configure time.

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used
            The problem here is that there is no way with ./configure to select the system libevent. This is needed for building .deb/.rpm for inclusion in distros and repositories.

            Libevent should work like the other external dependencies (libz, libreadline, libssl) where one can at configure time select between bundled and system version.

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used The problem here is that there is no way with ./configure to select the system libevent. This is needed for building .deb/.rpm for inclusion in distros and repositories. Libevent should work like the other external dependencies (libz, libreadline, libssl) where one can at configure time select between bundled and system version.
            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used
            User pallaert on IRC Freenode #maria reported an issue that shows one of the
            problems with this.

            He is using MariaDB with the Pinba plugin: http://pinba.org/

            Pinba uses libevent internally.

            In this case, mysqld hangs during shutdown. Here are relevant parts of a stack
            trace when it hangs:

            Thread 5 (Thread 0x9488ab70 (LWP 18676)):
            #0 0xb78b1424 in __kernel_vsyscall ()
            #1 0x49858507 in syscall () from /lib/libc.so.6
            #2 0x0844cb6d in epoll_wait (epfd=15, events=0xaba2aaa8, maxevents=1023, timeout=-1) at epoll_sub.c:57
            #3 0x0844c637 in epoll_dispatch (base=0xaba2a8f8, arg=0xaba06828, tv=0x0) at epoll.c:195
            #4 0x08449cf0 in event_base_loop (base=0xaba2a8f8, flags=0) at event.c:500
            #5 0x08449a9d in event_base_dispatch (event_base=0xaba2a8f8) at event.c:386
            #6 0xb6068d8a in pinba_collector_main (arg=0x0) at main.cc:164
            #7 0x499168df in start_thread () from /lib/libpthread.so.0

            Thread 4 (Thread 0x94089b70 (LWP 18677)):
            #0 0xb78b1424 in __kernel_vsyscall ()
            #1 0x49815036 in nanosleep () from /lib/libc.so.6
            #2 0x49854b1d in usleep () from /lib/libc.so.6
            #3 0xb60672ae in pinba_stats_main (arg=0x0) at pool.cc:518
            #4 0x499168df in start_thread () from /lib/libpthread.so.0

            Thread 2 (Thread 0xb559eb70 (LWP 18690)):
            #0 0xb78b1424 in __kernel_vsyscall ()
            #1 0x49917f27 in pthread_join () from /lib/libpthread.so.0
            #2 0xb6056caf in pinba_engine_shutdown (p=0x0) at ha_pinba.cc:289
            #3 0x0837ad7f in ha_finalize_handlerton (plugin=0xaf25c70) at handler.cc:403
            #4 0x08433dc3 in plugin_deinitialize (plugin=0xaf25c70, ref_check=true) at sql_plugin.cc:828
            #5 0x08433feb in reap_plugins () at sql_plugin.cc:916
            #6 0x0843483d in plugin_shutdown () at sql_plugin.cc:1544
            #7 0x0824f5a5 in clean_up (print_message=true) at mysqld.cc:1328
            #8 0x082523b0 in unireg_end () at mysqld.cc:1254
            #9 0x08252ecb in kill_server (sig_ptr=0x0) at mysqld.cc:1189
            #10 0x08252ee2 in kill_server_thread (arg=0x938883a0) at mysqld.cc:1217
            #11 0x499168df in start_thread () from /lib/libpthread.so.0

            Thread 1 (Thread 0xb78906c0 (LWP 18653)):
            #0 0xb78b1424 in __kernel_vsyscall ()
            #1 0x4991b6e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
            #2 0x087012f4 in safe_cond_wait (cond=0x88b2dc0, mp=0x88b2640, file=0x8767d80 "mysqld.cc", line=4667) at thr_mutex.c:496
            #3 0x0825232c in main (argc=9, argv=0xbfdc5cb4) at mysqld.cc:4667

            Thread 2 is stuck in pinba_engine_shutdown(). This function does:

            pthread_cancel(collector_thread);
            pthread_join(collector_thread, NULL);

            The collector_thread is Thread 5 in the stacktrace. It is stuck in epoll_wait
            inside libevent.

            Applying the patch above to use system libevent for mysqld solves the problem,
            and there is no longer any hang during shutdown.

            When mysqld is linked statically with the bundled libevent, and the pinba
            plugin links to the system libevent.so (likely a different version), we get
            inside the mysqld process a strange mix of code from the two different
            libevent versions, which causes havoc of course and is likely the cause of the
            above problem.

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used User pallaert on IRC Freenode #maria reported an issue that shows one of the problems with this. He is using MariaDB with the Pinba plugin: http://pinba.org/ Pinba uses libevent internally. In this case, mysqld hangs during shutdown. Here are relevant parts of a stack trace when it hangs: Thread 5 (Thread 0x9488ab70 (LWP 18676)): #0 0xb78b1424 in __kernel_vsyscall () #1 0x49858507 in syscall () from /lib/libc.so.6 #2 0x0844cb6d in epoll_wait (epfd=15, events=0xaba2aaa8, maxevents=1023, timeout=-1) at epoll_sub.c:57 #3 0x0844c637 in epoll_dispatch (base=0xaba2a8f8, arg=0xaba06828, tv=0x0) at epoll.c:195 #4 0x08449cf0 in event_base_loop (base=0xaba2a8f8, flags=0) at event.c:500 #5 0x08449a9d in event_base_dispatch (event_base=0xaba2a8f8) at event.c:386 #6 0xb6068d8a in pinba_collector_main (arg=0x0) at main.cc:164 #7 0x499168df in start_thread () from /lib/libpthread.so.0 Thread 4 (Thread 0x94089b70 (LWP 18677)): #0 0xb78b1424 in __kernel_vsyscall () #1 0x49815036 in nanosleep () from /lib/libc.so.6 #2 0x49854b1d in usleep () from /lib/libc.so.6 #3 0xb60672ae in pinba_stats_main (arg=0x0) at pool.cc:518 #4 0x499168df in start_thread () from /lib/libpthread.so.0 Thread 2 (Thread 0xb559eb70 (LWP 18690)): #0 0xb78b1424 in __kernel_vsyscall () #1 0x49917f27 in pthread_join () from /lib/libpthread.so.0 #2 0xb6056caf in pinba_engine_shutdown (p=0x0) at ha_pinba.cc:289 #3 0x0837ad7f in ha_finalize_handlerton (plugin=0xaf25c70) at handler.cc:403 #4 0x08433dc3 in plugin_deinitialize (plugin=0xaf25c70, ref_check=true) at sql_plugin.cc:828 #5 0x08433feb in reap_plugins () at sql_plugin.cc:916 #6 0x0843483d in plugin_shutdown () at sql_plugin.cc:1544 #7 0x0824f5a5 in clean_up (print_message=true) at mysqld.cc:1328 #8 0x082523b0 in unireg_end () at mysqld.cc:1254 #9 0x08252ecb in kill_server (sig_ptr=0x0) at mysqld.cc:1189 #10 0x08252ee2 in kill_server_thread (arg=0x938883a0) at mysqld.cc:1217 #11 0x499168df in start_thread () from /lib/libpthread.so.0 Thread 1 (Thread 0xb78906c0 (LWP 18653)): #0 0xb78b1424 in __kernel_vsyscall () #1 0x4991b6e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x087012f4 in safe_cond_wait (cond=0x88b2dc0, mp=0x88b2640, file=0x8767d80 "mysqld.cc", line=4667) at thr_mutex.c:496 #3 0x0825232c in main (argc=9, argv=0xbfdc5cb4) at mysqld.cc:4667 Thread 2 is stuck in pinba_engine_shutdown(). This function does: pthread_cancel(collector_thread); pthread_join(collector_thread, NULL); The collector_thread is Thread 5 in the stacktrace. It is stuck in epoll_wait inside libevent. Applying the patch above to use system libevent for mysqld solves the problem, and there is no longer any hang during shutdown. When mysqld is linked statically with the bundled libevent, and the pinba plugin links to the system libevent.so (likely a different version), we get inside the mysqld process a strange mix of code from the two different libevent versions, which causes havoc of course and is likely the cause of the above problem.
            Hide
            knielsen Kristian Nielsen added a comment -

            Full gdb stacktrace from the hang.
            LPexportBug509521_stacktrace.txt

            Show
            knielsen Kristian Nielsen added a comment - Full gdb stacktrace from the hang. LPexportBug509521_stacktrace.txt
            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used
            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used
            The fix for this will be in MariadB 5.1.43

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used The fix for this will be in MariadB 5.1.43
            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used
            Hm, it seems a complication here is that some of the distros we build for (like Ubuntu 8.04 hardy) has too old system libevent. At least the /usr/include/evutil.h is missing (it's included in eg. 9.10 karmic).

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used Hm, it seems a complication here is that some of the distros we build for (like Ubuntu 8.04 hardy) has too old system libevent. At least the /usr/include/evutil.h is missing (it's included in eg. 9.10 karmic).
            Hide
            knielsen Kristian Nielsen added a comment -

            Re: external (system) libevent cannot be used
            We can use system libevent from debian6 "squeeze" onwards and Ubuntu 9.10 "karmic" onward, but
            on earlier releases we must use bundled, as system is too old.

            Show
            knielsen Kristian Nielsen added a comment - Re: external (system) libevent cannot be used We can use system libevent from debian6 "squeeze" onwards and Ubuntu 9.10 "karmic" onward, but on earlier releases we must use bundled, as system is too old.
            Hide
            ratzpo Rasmus Johansson added a comment -

            Launchpad bug id: 509521

            Show
            ratzpo Rasmus Johansson added a comment - Launchpad bug id: 509521

              People

              • Assignee:
                serg Sergei Golubchik
                Reporter:
                arkadiuszmiśkiewicz Arkadiusz Miśkiewicz
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: