Details

      Description

      When I set the connect_work_size in /etc/my.cnf.d/connect.cnf
      it seems to be set properly but is not working.
      It goes alright when I indicate it using the set global command.

      When I write inside /etc/my.cnf.d/connect.cnf below,

      connect_work_size=536870912
      

      I get in Maria,

      show variables like 'connect_work_size';
      +-------------------+-----------+
      | Variable_name     | Value     |
      +-------------------+-----------+
      | connect_work_size | 536870912 |
      +-------------------+-----------+
      1 row in set (0.00 sec)
      

      When inserting into a connect table (SQLServer2005 table on another server),
      I get the following error inside /var/lib/mysql/servername.err

      PlugSubAlloc: Not enough memory in Work area for request of 80 (used=67108792 free=72)
      

      The size used+free is 64M, whcih is the default connect_work_size.
      NOT THE SIZE SET IN MY CNF FILE!

      When I set in Maria using the command,

      set global connect_work_size=536870912;
      

      the error changes to a correct one as below.

      PlugSubAlloc: Not enough memory in Work area for request of 40 (used=536870888 free=24)
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              elenst Elena Stepanova added a comment -

              Takuya Aoki,

              As I understand, it's an absolute repetition of MDEV-6690. In future, please don't create a new bug report if you have new information on an issue previously closed as "Can't reproduce" or "Incomplete"; instead, please comment on the old one and if the information is sufficient to proceed, the issue will be re-open.
              It's more useful for everyone because it will keep the history of the previous investigation.

              This time, I'll keep this one.

              Olivier Bertrand,

              I can easily reproduce it. The cnf file is not important, you can just as well set the option on the command line.

              First, I've made the trace printing unconditional, like this:

              === modified file 'storage/connect/plugutil.c'
              --- storage/connect/plugutil.c	2014-04-19 09:11:30 +0000
              +++ storage/connect/plugutil.c	2014-11-12 16:33:51 +0000
              @@ -490,6 +490,9 @@
                   htrc("SubAlloc in %p size=%d used=%d free=%d\n",
                         memp, size, pph->To_Free, pph->FreeBlk);
               
              +    if (trace)
              +      htrc("HERE: PlugSubAlloc: used=%d free=%d\n", pph->To_Free, pph->FreeBlk);
              +
                 if ((uint)size > pph->FreeBlk) {   /* Not enough memory left in pool */
                   char     *pname = "Work";
               
              

              Then I ran the server like this, with the non-default connect_work_size and with connect_xtrace:

              /home/elenst/bzr/10.0-bug/sql/mysqld --no-defaults --basedir=/10.0-bug --datadir=/10.0-bug/data --loose-lc-messages-dir=/10.0-bug/sql/share --loose-language=/10.0-bug/sql/share/english --port=3306 --socket=/10.0-bug/data/tmp/mysql.sock --plugin-load=ha_connect --console --connect_xtrace=2 --connect_work_size=536870912
              

              Then I connected to the server and created a simple Connect table:

              MariaDB [test]> select @@connect_work_size;
              +---------------------+
              | @@connect_work_size |
              +---------------------+
              |           536870912 |
              +---------------------+
              1 row in set (0.00 sec)
              
              MariaDB [test]> create table t1 (i int) engine=connect;
              Query OK, 0 rows affected, 2 warnings (0.24 sec)
              

              And here is what the trace told me:

              New CONNECT 0x7f24f5422ad8, table: <null>
              New CONNECT 0x7f24f542a088, table: t1
              Delete CONNECT 0x7f24f542a088, table: <null>, xp=(nil) count=0
              New CONNECT 0x7f24f542aa88, table: t1
              PlugInit: Language='Null'
              Memory of 67108864 allocated at 0x7f24f0400000
              Memory of 144 allocated at 0x7f24f54160c0
              create: this=0x7f24f542aa88 thd=0x7f2506319070 xp=0x7f24f54140e0 g=0x7f24f5448000 sqlcom=1 name=t1
              HERE: PlugSubAlloc: used=8 free=67108856
              after _splitpath: FileName=t1.dos
              dir= fname=t1 ext=.dos
              newname='./test/t1.dos'
              pbuff='/data/repo/bzr/10.0-bug/data/./test/t1.dos'
              xchk=(nil) createas=0
              Delete CONNECT 0x7f24f542aa88, table: <null>, xp=0x7f24f54140e0 count=1
              Delete CONNECT 0x7f24f5422ad8, table: <null>, xp=(nil) count=0
              

              Note that both the additional output and Memory of ... allocated indicate 67108864.

              Then I ran

              MariaDB [test]> set global connect_work_size =  536870912;
              Query OK, 0 rows affected (0.00 sec)
              
              MariaDB [test]> select @@connect_work_size;
              +---------------------+
              | @@connect_work_size |
              +---------------------+
              |           536870912 |
              +---------------------+
              1 row in set (0.00 sec)
              
              MariaDB [test]> create table t2 (i int) engine=connect;
              Query OK, 0 rows affected, 2 warnings (0.17 sec)
              

              And got this:

              New CONNECT 0x7f24f5422ad8, table: <null>
              New CONNECT 0x7f24f542b488, table: t2
              Delete CONNECT 0x7f24f542b488, table: <null>, xp=(nil) count=0
              New CONNECT 0x7f24f542a088, table: t2
              PlugInit: Language='Null'
              Memory of 536870912 allocated at 0x7f24d0400000
              Memory of 144 allocated at 0x7f24f54160c0
              create: this=0x7f24f542a088 thd=0x7f2506319070 xp=0x7f24f54140e0 g=0x7f24f5448000 sqlcom=1 name=t2
              HERE: PlugSubAlloc: used=8 free=536870904
              after _splitpath: FileName=t2.dos
              dir= fname=t2 ext=.dos
              newname='./test/t2.dos'
              pbuff='/data/repo/bzr/10.0-bug/data/./test/t2.dos'
              xchk=(nil) createas=0
              Delete CONNECT 0x7f24f542a088, table: <null>, xp=0x7f24f54140e0 count=1
              Delete CONNECT 0x7f24f5422ad8, table: <null>, xp=(nil) count=0
              

              Note that now both the additional and the standard printing show 536870912.

              Now, I'm not familiar with the code to debug it further, but I read in MDEV-6690 that you debugged it before and found that internally the value works correctly; if so, maybe it's just the values that are used for output are set incorrectly.

              Show
              elenst Elena Stepanova added a comment - Takuya Aoki , As I understand, it's an absolute repetition of MDEV-6690 . In future, please don't create a new bug report if you have new information on an issue previously closed as "Can't reproduce" or "Incomplete"; instead, please comment on the old one and if the information is sufficient to proceed, the issue will be re-open. It's more useful for everyone because it will keep the history of the previous investigation. This time, I'll keep this one. Olivier Bertrand , I can easily reproduce it. The cnf file is not important, you can just as well set the option on the command line. First, I've made the trace printing unconditional, like this: === modified file 'storage/connect/plugutil.c' --- storage/connect/plugutil.c 2014-04-19 09:11:30 +0000 +++ storage/connect/plugutil.c 2014-11-12 16:33:51 +0000 @@ -490,6 +490,9 @@ htrc("SubAlloc in %p size=%d used=%d free=%d\n", memp, size, pph->To_Free, pph->FreeBlk); + if (trace) + htrc("HERE: PlugSubAlloc: used=%d free=%d\n", pph->To_Free, pph->FreeBlk); + if ((uint)size > pph->FreeBlk) { /* Not enough memory left in pool */ char *pname = "Work"; Then I ran the server like this, with the non-default connect_work_size and with connect_xtrace: /home/elenst/bzr/10.0-bug/sql/mysqld --no-defaults --basedir=/10.0-bug --datadir=/10.0-bug/data --loose-lc-messages-dir=/10.0-bug/sql/share --loose-language=/10.0-bug/sql/share/english --port=3306 --socket=/10.0-bug/data/tmp/mysql.sock --plugin-load=ha_connect --console --connect_xtrace=2 --connect_work_size=536870912 Then I connected to the server and created a simple Connect table: MariaDB [test]> select @@connect_work_size; +---------------------+ | @@connect_work_size | +---------------------+ | 536870912 | +---------------------+ 1 row in set (0.00 sec) MariaDB [test]> create table t1 (i int) engine=connect; Query OK, 0 rows affected, 2 warnings (0.24 sec) And here is what the trace told me: New CONNECT 0x7f24f5422ad8, table: <null> New CONNECT 0x7f24f542a088, table: t1 Delete CONNECT 0x7f24f542a088, table: <null>, xp=(nil) count=0 New CONNECT 0x7f24f542aa88, table: t1 PlugInit: Language='Null' Memory of 67108864 allocated at 0x7f24f0400000 Memory of 144 allocated at 0x7f24f54160c0 create: this=0x7f24f542aa88 thd=0x7f2506319070 xp=0x7f24f54140e0 g=0x7f24f5448000 sqlcom=1 name=t1 HERE: PlugSubAlloc: used=8 free=67108856 after _splitpath: FileName=t1.dos dir= fname=t1 ext=.dos newname='./test/t1.dos' pbuff='/data/repo/bzr/10.0-bug/data/./test/t1.dos' xchk=(nil) createas=0 Delete CONNECT 0x7f24f542aa88, table: <null>, xp=0x7f24f54140e0 count=1 Delete CONNECT 0x7f24f5422ad8, table: <null>, xp=(nil) count=0 Note that both the additional output and Memory of ... allocated indicate 67108864. Then I ran MariaDB [test]> set global connect_work_size = 536870912; Query OK, 0 rows affected (0.00 sec) MariaDB [test]> select @@connect_work_size; +---------------------+ | @@connect_work_size | +---------------------+ | 536870912 | +---------------------+ 1 row in set (0.00 sec) MariaDB [test]> create table t2 (i int) engine=connect; Query OK, 0 rows affected, 2 warnings (0.17 sec) And got this: New CONNECT 0x7f24f5422ad8, table: <null> New CONNECT 0x7f24f542b488, table: t2 Delete CONNECT 0x7f24f542b488, table: <null>, xp=(nil) count=0 New CONNECT 0x7f24f542a088, table: t2 PlugInit: Language='Null' Memory of 536870912 allocated at 0x7f24d0400000 Memory of 144 allocated at 0x7f24f54160c0 create: this=0x7f24f542a088 thd=0x7f2506319070 xp=0x7f24f54140e0 g=0x7f24f5448000 sqlcom=1 name=t2 HERE: PlugSubAlloc: used=8 free=536870904 after _splitpath: FileName=t2.dos dir= fname=t2 ext=.dos newname='./test/t2.dos' pbuff='/data/repo/bzr/10.0-bug/data/./test/t2.dos' xchk=(nil) createas=0 Delete CONNECT 0x7f24f542a088, table: <null>, xp=0x7f24f54140e0 count=1 Delete CONNECT 0x7f24f5422ad8, table: <null>, xp=(nil) count=0 Note that now both the additional and the standard printing show 536870912. Now, I'm not familiar with the code to debug it further, but I read in MDEV-6690 that you debugged it before and found that internally the value works correctly; if so, maybe it's just the values that are used for output are set incorrectly.
              Hide
              takuya Takuya Aoki added a comment -

              It wasn't a good idea to make a report again, but this is a crucial problem so I am asking fo further consideration.
              I have to use the set global command if I want the proper connect_work_size which is a hesitation.

              Anyway, thank you for reproducing the bug.
              I understand you are getting 67108864, which conflicts with the connect_work_size set at 536870912.
              There is a misunderstanding that I debugged it.
              I have no idea how to make the cnf file work so that I can work with the right connect_work_size.
              I think this is a program error since other system variables like the ones in the below site can be set properly in the cnf files and works.
              https://mariadb.com/kb/en/mariadb/documentation/optimization-and-tuning/system-variables/server-system-variables/

              Show
              takuya Takuya Aoki added a comment - It wasn't a good idea to make a report again, but this is a crucial problem so I am asking fo further consideration. I have to use the set global command if I want the proper connect_work_size which is a hesitation. Anyway, thank you for reproducing the bug. I understand you are getting 67108864, which conflicts with the connect_work_size set at 536870912. There is a misunderstanding that I debugged it. I have no idea how to make the cnf file work so that I can work with the right connect_work_size. I think this is a program error since other system variables like the ones in the below site can be set properly in the cnf files and works. https://mariadb.com/kb/en/mariadb/documentation/optimization-and-tuning/system-variables/server-system-variables/
              Hide
              elenst Elena Stepanova added a comment -

              Takuya Aoki,

              There is a misunderstanding that I debugged it.

              It was a part addressed to Olivier – I meant that he analyzed it before.

              Show
              elenst Elena Stepanova added a comment - Takuya Aoki , There is a misunderstanding that I debugged it. It was a part addressed to Olivier – I meant that he analyzed it before.
              Hide
              takuya Takuya Aoki added a comment -

              Elena Stepanova,

              I understand now, Thank you.
              This is my opinion but I don't think testing on Windows as in MDEV 6690 is a good idea, since the bug might depend on the environment.

              Show
              takuya Takuya Aoki added a comment - Elena Stepanova, I understand now, Thank you. This is my opinion but I don't think testing on Windows as in MDEV 6690 is a good idea, since the bug might depend on the environment.
              Hide
              bertrandop Olivier Bertrand added a comment - - edited

              I found why this happened. BTW it should also happen when the size is given as a command line option when starting the server.

              It is both a CONNECT bug and an issue of MySQL/MariaDB system variable handling. In the CONNECT code, the work area is allocated at the very beginning according to an internal variable "worksize". This was hard coded but when I introduced the possibility to change it via the connect_work_size variable I did not change the way this was done but simply added an "update" function to the connect_work_size definition in which I changed the value of static worksize variable according to the new value given to the connect_work_size variable.

              The issue is that this "update" function is called when using the SET command but is not called to set the initial value coming from the cnf file or the command line. Therefore the system variable had the good value but the internal variable was not set.

              I shall fix this by using the work_size global variable directly instead of setting another variable, which in fact simplifies the job as the use of that "update" function will no longer be required.

              However, I still regard this as a MariaDB issue (bug?) "update" and "check" functions are there to check the values given to system variables and this should also apply to values given va the cnf file and/or command line options. These functions should be called in that case.

              It would also enables things such as giving the work size as 512M in the SET command. The variable would be declared as "string" insread of "integer" and in the "update" function it would be possible to change the representation to an integer, the same that is done when analysing the values given in cnf file.

              Show
              bertrandop Olivier Bertrand added a comment - - edited I found why this happened. BTW it should also happen when the size is given as a command line option when starting the server. It is both a CONNECT bug and an issue of MySQL/MariaDB system variable handling. In the CONNECT code, the work area is allocated at the very beginning according to an internal variable "worksize". This was hard coded but when I introduced the possibility to change it via the connect_work_size variable I did not change the way this was done but simply added an "update" function to the connect_work_size definition in which I changed the value of static worksize variable according to the new value given to the connect_work_size variable. The issue is that this "update" function is called when using the SET command but is not called to set the initial value coming from the cnf file or the command line. Therefore the system variable had the good value but the internal variable was not set. I shall fix this by using the work_size global variable directly instead of setting another variable, which in fact simplifies the job as the use of that "update" function will no longer be required. However, I still regard this as a MariaDB issue (bug?) "update" and "check" functions are there to check the values given to system variables and this should also apply to values given va the cnf file and/or command line options. These functions should be called in that case. It would also enables things such as giving the work size as 512M in the SET command. The variable would be declared as "string" insread of "integer" and in the "update" function it would be possible to change the representation to an integer, the same that is done when analysing the values given in cnf file.
              Hide
              elenst Elena Stepanova added a comment -

              Olivier Bertrand,

              Could you please file a separate bug for the flaw that you found in MariaDB variable handling?

              Thanks.

              And yes, as I said in an above comment, it does happen when the option is given on the command line.

              Show
              elenst Elena Stepanova added a comment - Olivier Bertrand , Could you please file a separate bug for the flaw that you found in MariaDB variable handling? Thanks. And yes, as I said in an above comment, it does happen when the option is given on the command line.
              Hide
              takuya Takuya Aoki added a comment -

              Was this bug fixed in version 10.0.16?
              Since the bug related to it was fixed, I have no way of checking.

              Show
              takuya Takuya Aoki added a comment - Was this bug fixed in version 10.0.16? Since the bug related to it was fixed, I have no way of checking.

                People

                • Assignee:
                  bertrandop Olivier Bertrand
                  Reporter:
                  takuya Takuya Aoki
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 4 hours
                    4h