Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-4889

Race condition on shutdown when threadpool is used

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Minor
        • Resolution: Cannot Reproduce
        • Affects Version/s: 10.0.3, 5.5.32
        • Fix Version/s: 10.0.5, 5.5.33
        • Component/s: None
        • Labels:
          None

          Description

          All invocations of post_kill_notification are protected by LOCK_thd_data except one in close_connections. This may lead to race condition. post_kill_notification will shutdown the socket, event handler thread will receive notification and will destroy the THD at the same time.

            Gliffy Diagrams

              Attachments

                Issue Links

                links to

                Web Link *** glibc detected *** bin/mysqld: free(): corrupted unsorted chunks: 0x000000000321c4a0 *** AT SHUTDOWN | handle_fatal_signal (sig=6) in _int_free (libc) from Events::deinit at sql/events.cc:930

                  Activity

                  Ascending order - Click to sort in descending order
                  Hide
                  Permalink
                  laurynas Laurynas Biveinis added a comment -

                  https://bugs.launchpad.net/bugs/1206065

                  Show
                  laurynas Laurynas Biveinis added a comment - https://bugs.launchpad.net/bugs/1206065
                  Hide
                  Permalink
                  wlad Vladislav Vaintroub added a comment - - edited

                  Is there any evidence that MariaDB (not Percona Server) has this bug?

                  In MariaDB, freeing memory (vio_delete) happens when THD destructor is running.. At this moment, THD is not anymore in the global threads list (see unlink_thd() calling thd->unlink() before THD destructor). The "unlink" operation is protected by LOCK_thread_count. Therefore, code in server shutdown, that traverses the threads lists under protection of the same LOCK_thread_count mutex , and issues post_kill_notification, this code cannot possibly access freed memory.

                  In Percona Server 5.6, on the other hand, accessing freed memory is quite possible . Freeing vio memory happens under in threadpool_remove_connection in thd->release_resources(), which is newly introduced in 5.6. MariaDB does not have THD::release_resources, and the threadpool_remove_connection is the same as in 5.5, with proper LOCK_thread_count mutex protection as described above.

                  Show
                  wlad Vladislav Vaintroub added a comment - - edited Is there any evidence that MariaDB (not Percona Server) has this bug? In MariaDB, freeing memory (vio_delete) happens when THD destructor is running.. At this moment, THD is not anymore in the global threads list (see unlink_thd() calling thd->unlink() before THD destructor). The "unlink" operation is protected by LOCK_thread_count. Therefore, code in server shutdown, that traverses the threads lists under protection of the same LOCK_thread_count mutex , and issues post_kill_notification, this code cannot possibly access freed memory. In Percona Server 5.6, on the other hand, accessing freed memory is quite possible . Freeing vio memory happens under in threadpool_remove_connection in thd->release_resources(), which is newly introduced in 5.6. MariaDB does not have THD::release_resources, and the threadpool_remove_connection is the same as in 5.5, with proper LOCK_thread_count mutex protection as described above.
                  Hide
                  Permalink
                  wlad Vladislav Vaintroub added a comment - - edited

                  As for Percona Server, I think that that newly introduced thd->release_resources(); is the culprit, and that it is not required at all (because it will be called anyway in THD destructor, after the THD is removed from the global list)

                  Show
                  wlad Vladislav Vaintroub added a comment - - edited As for Percona Server, I think that that newly introduced thd->release_resources(); is the culprit, and that it is not required at all (because it will be called anyway in THD destructor, after the THD is removed from the global list)
                  Hide
                  Permalink
                  wlad Vladislav Vaintroub added a comment -

                  Closing for now, since the bug is non-existent in MariaDB. Please feel free to reopen if I missed something in the analysis above.

                  Show
                  wlad Vladislav Vaintroub added a comment - Closing for now, since the bug is non-existent in MariaDB. Please feel free to reopen if I missed something in the analysis above.

                    People

                    • Assignee:
                      wlad Vladislav Vaintroub
                      Reporter:
                      sergei Sergei Glushchenko
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      4 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: