Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6531

MySQL Bug #72809 - InnoDB Linux native aio setup missing barrier after setup

        Details

        • Type: Task
        • Status: Stalled
        • Priority: Major
        • Resolution: Unresolved
        • Fix Version/s: 10.0
        • Component/s: None
        • Labels:
          None

          Description

          The following patch is suggested by Stewart Smith:

          memory barrier at end of setting up innodb aio

we need a memory barrier here as otherwise when IO threads start they
can get a stale view of the data structure.

It turns out that POWER8 is fast enough with large enough caches that
this can actually be a problem.

You can reproduce this by attempting to do a lot of writes shortly after
server startup - you'll hit a really odd assert.

Index: mysql-5.6.17/storage/innobase/os/os0file.cc
===================================================================
--- mysql-5.6.17.orig/storage/innobase/os/os0file.cc
+++ mysql-5.6.17/storage/innobase/os/os0file.cc
@@ -3782,7 +3782,7 @@ skip_native_aio:
                slot->ret = 0;
 #endif /* WIN_ASYNC_IO */
        }
-
+       __sync_synchronize();
        return(array);
 }

            Gliffy Diagrams

              Attachments

                Issue Links

                is part of

                Task - A task that needs to be done. MDEV-6530 Examine and apply Power8 patches suggested by Stewart Smith

                • Major - Major loss of function.
                • Open
                links to

                Web Link Kernel patch which may be relevant

                Web Link MySQL BUG#72809

                  Activity

                  Ascending order - Click to sort in descending order
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment - - edited

                  Quoting Stewart:

                  Basically, as InnoDB sets up the data structures for aio during startup, it creates a mutex but doesn't take it, thus not issuing any memory barriers.

                  It turns out that a POWER8 seems to be fast enough and with enough caches that you can get into runtime inside InnoDB with InnoDB doing IO where one of the InnoDB IO threads will hit an assert due to reading stale data left over from setting up AIO rather than runtime.

                  This patch adds __sync_synchronize() to
                  innobase_init()/innobase_start_or_create_for_mysql()/os_aio_init()/os_aio_array_create()

                  But I/O threads are created later in
                  innobase_init()/innobase_start_or_create_for_mysql()

                  And pthread_create() is supposed to synchronize memory anyway:
                  http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_11

                  Show
                  svoj Sergey Vojtovich added a comment - - edited Quoting Stewart: Basically, as InnoDB sets up the data structures for aio during startup, it creates a mutex but doesn't take it, thus not issuing any memory barriers. It turns out that a POWER8 seems to be fast enough and with enough caches that you can get into runtime inside InnoDB with InnoDB doing IO where one of the InnoDB IO threads will hit an assert due to reading stale data left over from setting up AIO rather than runtime. This patch adds __sync_synchronize() to innobase_init()/innobase_start_or_create_for_mysql()/os_aio_init()/os_aio_array_create() But I/O threads are created later in innobase_init()/innobase_start_or_create_for_mysql() And pthread_create() is supposed to synchronize memory anyway: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_11
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Closing as can't repeat for now. We will reopen this issue if we will be able to reproduce anything similar.

                  Show
                  svoj Sergey Vojtovich added a comment - Closing as can't repeat for now. We will reopen this issue if we will be able to reproduce anything similar.
                  Hide
                  Permalink
                  axel Axel Schwenke added a comment -

                  will try to reproduce

                  Show
                  axel Axel Schwenke added a comment - will try to reproduce

                    People

                    • Assignee:
                      axel Axel Schwenke
                      Reporter:
                      svoj Sergey Vojtovich
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      3 Start watching this issue

                      Dates

                      • Created:
                        Updated: