Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-102

mysql-test-run runs too slow and/or consumes too much disk space and memory

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.5.20
    • Fix Version/s: 5.5.20
    • Component/s: None
    • Labels:

      Description

      mysql-test-run consumes much more resources in 5.5 compared to 5.3.

      In 5.3 I was able to run the entire test suite (with --mem --parallel=10) in around 6 minutes on a machine with 8GB of memory. 3 min for a release build.

      In 5.5 it takes much longer. And worse, we see failures on hosts due to consuming too much resource. Eg. the DGCov builder in buildbot ran out of memory just using --mem --parallel=2, with 3GB of memory available.

      It still needs to be quantified more precisely what the problems are. Possibilities include

      • Writing too large files in mysql-test/var/, overflowing memory
      • safemalloc overhead
      • unit tests being included in mtr (f.ex. some of the aria unit tests consume huge amount of resources)
      • something else entirely

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            knielsen Kristian Nielsen added a comment -

            For example, after I reduced DGCov in buildbot to --parallel=2 and no --mem, the test works, but it takes around 10 hours to complete, which is hardly acceptable.

            Show
            knielsen Kristian Nielsen added a comment - For example, after I reduced DGCov in buildbot to --parallel=2 and no --mem, the test works, but it takes around 10 hours to complete, which is hardly acceptable.
            Hide
            wlad Vladislav Vaintroub added a comment -

            Please ignore all the changes I done to that bug so far.Mixed something up in JIRA. I'm not working on this one.

            Show
            wlad Vladislav Vaintroub added a comment - Please ignore all the changes I done to that bug so far.Mixed something up in JIRA. I'm not working on this one.
            Hide
            serg Sergei Golubchik added a comment -

            I'm not sure we're talking about the same buildbot and the same dgcov.

            http://buildbot.askmonty.org/buildbot/builders/kvm-dgcov-jaunty-i386/builds/1097
            5.3 build.
            test nm: 2 hrs, 43 mins, 33 secs
            total elapsed (compiling, gcov, few test runs): 6 hrs, 53 mins, 31 secs

            http://buildbot.askmonty.org/buildbot/builders/kvm-dgcov-jaunty-i386/builds/1094

            5.5 build
            test nm: 2 hrs, 43 mins, 37 secs
            total elapsed: 6 hrs, 18 mins, 51 secs

            Other builders (that I've checked) also show that in 5.5 tests don't take significantly more time that in 5.3

            Show
            serg Sergei Golubchik added a comment - I'm not sure we're talking about the same buildbot and the same dgcov. http://buildbot.askmonty.org/buildbot/builders/kvm-dgcov-jaunty-i386/builds/1097 5.3 build. test nm: 2 hrs, 43 mins, 33 secs total elapsed (compiling, gcov, few test runs): 6 hrs, 53 mins, 31 secs http://buildbot.askmonty.org/buildbot/builders/kvm-dgcov-jaunty-i386/builds/1094 5.5 build test nm: 2 hrs, 43 mins, 37 secs total elapsed: 6 hrs, 18 mins, 51 secs Other builders (that I've checked) also show that in 5.5 tests don't take significantly more time that in 5.3
            Hide
            serg Sergei Golubchik added a comment -

            ok. tests are only slow if --mem is used. They consume all available memory.
            Because in 5.5 unit tests are part of mtr run, and few aria unit tests create huge files:

            ma_pagecache_single_8k-t => 160 Mb
            ma_test_loghandler_multithread-t => 692 Mb
            ma_test_loghandler_multigroup-t => 1189 Mb
            ma_pagecache_single_64k-t => 1280 Mb
            ma_test_loghandler_multithread-t => 1536 Mb

            Show
            serg Sergei Golubchik added a comment - ok. tests are only slow if --mem is used. They consume all available memory. Because in 5.5 unit tests are part of mtr run, and few aria unit tests create huge files: ma_pagecache_single_8k-t => 160 Mb ma_test_loghandler_multithread-t => 692 Mb ma_test_loghandler_multigroup-t => 1189 Mb ma_pagecache_single_64k-t => 1280 Mb ma_test_loghandler_multithread-t => 1536 Mb
            Hide
            serg Sergei Golubchik added a comment -

            although it's easy to skip the complete test as "big", it would be better to adjust the test size to create smaller files, unless "big" test is requested (that is, skip_big_tests is 0).

            when fixing it, one also needs to fix suite/unit/suite.pm to pass "big" option down to test executables:

            $ENV

            {MYTAP_CONFIG}

            ='big' if $::opt_big_test;

            Show
            serg Sergei Golubchik added a comment - although it's easy to skip the complete test as "big", it would be better to adjust the test size to create smaller files, unless "big" test is requested (that is, skip_big_tests is 0). when fixing it, one also needs to fix suite/unit/suite.pm to pass "big" option down to test executables: $ENV {MYTAP_CONFIG} ='big' if $::opt_big_test;
            Hide
            sanja Oleksandr Byelkin added a comment -

            It has no sens to run that test on smaller files (nothing to test). so I'll just skip it on BIG. But have we at least one buildbot slave which runs "big" test?

            Show
            sanja Oleksandr Byelkin added a comment - It has no sens to run that test on smaller files (nothing to test). so I'll just skip it on BIG. But have we at least one buildbot slave which runs "big" test?
            Hide
            sanja Oleksandr Byelkin added a comment -

            === modified file 'storage/maria/unittest/CMakeLists.txt'
            — storage/maria/unittest/CMakeLists.txt 2011-11-22 17:04:38 +0000
            +++ storage/maria/unittest/CMakeLists.txt 2012-01-27 08:24:08 +0000
            @@ -76,12 +76,12 @@ MY_ADD_TEST(ma_pagecache_single_1k)

            ADD_EXECUTABLE(ma_pagecache_single_8k-t $

            {ma_pagecache_single_src})
            SET_TARGET_PROPERTIES(ma_pagecache_single_8k-t
            - PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=8192")
            + PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=8192 -DBIG")
            MY_ADD_TEST(ma_pagecache_single_8k)

            ADD_EXECUTABLE(ma_pagecache_single_64k-t ${ma_pagecache_single_src}

            )
            SET_TARGET_PROPERTIES(ma_pagecache_single_64k-t

            • PROPERTIES COMPILE_FLAGS "$ {ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=65536")
              + PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags}

              -DTEST_PAGE_SIZE=65536 -DBIG")
              MY_ADD_TEST(ma_pagecache_single_64k)

            ADD_EXECUTABLE(ma_pagecache_consist_1k-t $

            {ma_pagecache_consist_src}

            )

            === modified file 'storage/maria/unittest/ma_test_loghandler_multigroup-t.c'
            — storage/maria/unittest/ma_test_loghandler_multigroup-t.c 2011-10-19 19:45:18 +0000
            +++ storage/maria/unittest/ma_test_loghandler_multigroup-t.c 2012-01-27 08:53:46 +0000
            @@ -39,6 +39,7 @@ static TRN *trn= &dummy_transaction_obje
            #define LOG_FILE_SIZE (1024L*1024L*1024L + 1024L*1024L*512)
            #define ITERATIONS 2
            #define READONLY 0
            +#define BIG 1

            #else

            @@ -48,6 +49,7 @@ static TRN *trn= &dummy_transaction_obje
            #define LOG_FILE_SIZE (1024L*1024L*1024L + 1024L*1024L*512)
            #define ITERATIONS 2
            #define READONLY 1
            +#undef BIG

            #endif /READONLY_TEST/

            @@ -243,6 +245,15 @@ int main(int argc _attribute_((unused)
            int rc;
            MY_INIT(argv[0]);

            + plan(0); // read configuration (MYTAP_CONFIG)
            +#ifdef BIG
            + if (skip_big_tests)
            +

            { + plan(1); + ok(1, "skipped as big test"); + return 0; + }

            +#endif

            load_defaults("my", load_default_groups, &argc, &argv);
            get_options(&argc, &argv);

            === modified file 'storage/maria/unittest/ma_test_loghandler_multithread-t.c'
            — storage/maria/unittest/ma_test_loghandler_multithread-t.c 2011-10-19 19:45:18 +0000
            +++ storage/maria/unittest/ma_test_loghandler_multithread-t.c 2012-01-27 08:55:11 +0000
            @@ -35,7 +35,7 @@ static const char *default_dbug_option;

            #ifdef MULTIFLUSH_TEST

            -#define LONG_BUFFER_SIZE (16384L)
            +#define LONG_BUFFER_SZ (16384L)
            #define MIN_REC_LENGTH 10
            #define SHOW_DIVIDER 20
            #define ITERATIONS 10000
            @@ -45,7 +45,7 @@ static const char *default_dbug_option;

            #else

            -#define LONG_BUFFER_SIZE (512L*1024L*1024L)
            +#define LONG_BUFFER_SZ (512L*1024L*1024L)
            #define MIN_REC_LENGTH 30
            #define SHOW_DIVIDER 10
            #define ITERATIONS 3
            @@ -55,6 +55,8 @@ static const char *default_dbug_option;

            #endif

            +#define LONG_BUFFER_SIZE (LONG_BUFFER_SZ >> (skip_big_tests ? 4 : 0))
            +
            static uint number_of_writers= WRITERS;
            static uint number_of_flushers= FLUSHERS;

            @@ -270,6 +272,7 @@ int main(int argc _attribute_((unused)
            int rc;
            MY_INIT(argv[0]);

            + // plan read MYTAP_CONFIG so skip_big_tests will be set before using
            plan(WRITERS + FLUSHERS +
            ITERATIONS * WRITERS * 3 + FLUSH_ITERATIONS * FLUSHERS );
            /* We don't need to do physical syncs in this test */

            Show
            sanja Oleksandr Byelkin added a comment - === modified file 'storage/maria/unittest/CMakeLists.txt' — storage/maria/unittest/CMakeLists.txt 2011-11-22 17:04:38 +0000 +++ storage/maria/unittest/CMakeLists.txt 2012-01-27 08:24:08 +0000 @@ -76,12 +76,12 @@ MY_ADD_TEST(ma_pagecache_single_1k) ADD_EXECUTABLE(ma_pagecache_single_8k-t $ {ma_pagecache_single_src}) SET_TARGET_PROPERTIES(ma_pagecache_single_8k-t - PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=8192") + PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=8192 -DBIG") MY_ADD_TEST(ma_pagecache_single_8k) ADD_EXECUTABLE(ma_pagecache_single_64k-t ${ma_pagecache_single_src} ) SET_TARGET_PROPERTIES(ma_pagecache_single_64k-t PROPERTIES COMPILE_FLAGS "$ {ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=65536") + PROPERTIES COMPILE_FLAGS "${ma_pagecache_common_cppflags} -DTEST_PAGE_SIZE=65536 -DBIG") MY_ADD_TEST(ma_pagecache_single_64k) ADD_EXECUTABLE(ma_pagecache_consist_1k-t $ {ma_pagecache_consist_src} ) === modified file 'storage/maria/unittest/ma_test_loghandler_multigroup-t.c' — storage/maria/unittest/ma_test_loghandler_multigroup-t.c 2011-10-19 19:45:18 +0000 +++ storage/maria/unittest/ma_test_loghandler_multigroup-t.c 2012-01-27 08:53:46 +0000 @@ -39,6 +39,7 @@ static TRN *trn= &dummy_transaction_obje #define LOG_FILE_SIZE (1024L*1024L*1024L + 1024L*1024L*512) #define ITERATIONS 2 #define READONLY 0 +#define BIG 1 #else @@ -48,6 +49,7 @@ static TRN *trn= &dummy_transaction_obje #define LOG_FILE_SIZE (1024L*1024L*1024L + 1024L*1024L*512) #define ITERATIONS 2 #define READONLY 1 +#undef BIG #endif / READONLY_TEST / @@ -243,6 +245,15 @@ int main(int argc _ attribute _((unused) int rc; MY_INIT(argv [0] ); + plan(0); // read configuration (MYTAP_CONFIG) +#ifdef BIG + if (skip_big_tests) + { + plan(1); + ok(1, "skipped as big test"); + return 0; + } +#endif load_defaults("my", load_default_groups, &argc, &argv); get_options(&argc, &argv); === modified file 'storage/maria/unittest/ma_test_loghandler_multithread-t.c' — storage/maria/unittest/ma_test_loghandler_multithread-t.c 2011-10-19 19:45:18 +0000 +++ storage/maria/unittest/ma_test_loghandler_multithread-t.c 2012-01-27 08:55:11 +0000 @@ -35,7 +35,7 @@ static const char *default_dbug_option; #ifdef MULTIFLUSH_TEST -#define LONG_BUFFER_SIZE (16384L) +#define LONG_BUFFER_SZ (16384L) #define MIN_REC_LENGTH 10 #define SHOW_DIVIDER 20 #define ITERATIONS 10000 @@ -45,7 +45,7 @@ static const char *default_dbug_option; #else -#define LONG_BUFFER_SIZE (512L*1024L*1024L) +#define LONG_BUFFER_SZ (512L*1024L*1024L) #define MIN_REC_LENGTH 30 #define SHOW_DIVIDER 10 #define ITERATIONS 3 @@ -55,6 +55,8 @@ static const char *default_dbug_option; #endif +#define LONG_BUFFER_SIZE (LONG_BUFFER_SZ >> (skip_big_tests ? 4 : 0)) + static uint number_of_writers= WRITERS; static uint number_of_flushers= FLUSHERS; @@ -270,6 +272,7 @@ int main(int argc _ attribute _((unused) int rc; MY_INIT(argv [0] ); + // plan read MYTAP_CONFIG so skip_big_tests will be set before using plan(WRITERS + FLUSHERS + ITERATIONS * WRITERS * 3 + FLUSH_ITERATIONS * FLUSHERS ); /* We don't need to do physical syncs in this test */
            Hide
            sanja Oleksandr Byelkin added a comment -

            fix pushed

            Show
            sanja Oleksandr Byelkin added a comment - fix pushed

              People

              • Assignee:
                sanja Oleksandr Byelkin
                Reporter:
                knielsen Kristian Nielsen
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Time Spent - 2 hours Remaining Estimate - 1 hour
                  1h
                  Logged:
                  Time Spent - 2 hours Remaining Estimate - 1 hour
                  2h