[MDEV-6552] TokuDB requires HUGE setting,... - JIRA

Details

Type: Bug
Status: Closed
Priority: Trivial
Resolution: Incomplete
Affects Version/s: 10.0.10-galera
Fix Version/s: N/A
Component/s: None
Labels:
None
Environment:
Ubuntu Tryusty

Description

hi,

I moved from 12.04 Ubuntu 14.04.
Installed Mariadb - galera from ditribution and moved old datbases and settings to new one. Old one was build from sources and not galera.

I had to add line:

exec "echo never >/sys/kernel/mm/transparent_hugepage/enabled"

before exec mysqld line at startup script to get mariadb functioning.

That's needed to get TokuDB functionin and maraiadb goin up.

Other abnormal behaviour faced were:

mysqld server did hang and prevented normal start and stop,
no error log messages , THAT WAS BEFORE SCRIPT CHANGE

after killing mysqld by KILL and got mysqld -v to show problem
I COULD NOT TERMINATE started mysqld -v on terminal whit Ctrl-X-C !
Complains that error.log is old format ! , WHILE GETTING PROBLEM SORTED DUE NO ERROR LOG

Gliffy Diagrams

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

my.cnf
5 kB
2014-08-08 18:23

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Joni-Pekka Kurronen added a comment - 2014-08-08 18:22 - edited

mysql < bacula.sql command run's abt. 8 hours ending TOTAL CRASH OF UBUNTU Trusty desktop whit 32GB ram, 1T RADI1 ( 2x1T ),... after

it it take 3 hours to sync RAID1 disk.

bacula.sql size is abt 3G and useing Tokudb

Log say's very litle:

/usr/sbin/mysqld, Version: 10.0.12-MariaDB-1~trusty-wsrep-log (mariadb.org binary distribution, wsrep_25.10.r4002). started with:
Tcp port: 3306  Unix socket: /var/run/mysqld/mysqld.sock
Time                 Id Command    Argument

This error can be reproduced but take's 8 hours.

Manualy started to starup, need's HUGE SETTING
before start.

joni@mpi1:~$ sudo su
root@mpi1:/home/joni# echo never >/sys/kernel/mm/transparent_hugepage/enabled
root@mpi1:/home/joni# exit
exit
joni@mpi1:~$ sudo mysqld
140808 18:35:02 [Warning] option 'table_definition_cache': unsigned value 100 adjusted to 400
140808 18:35:02 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/local/mysql/share/english/
140808 18:35:02 [Warning] Use --lc-messages-dir without language specific part instead.
140808 18:35:02 [ERROR] Error message file 'errmsg.sys' is probably from and older version of MariaDB / MYSQL as it doesn't contain all error messages
2014-08-08 18:35:02 7f61397067c0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
140808 18:35:02 [Note] InnoDB: Using mutexes to ref count buffer pool pages
140808 18:35:02 [Note] InnoDB: The InnoDB memory heap is disabled
140808 18:35:02 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
140808 18:35:02 [Note] InnoDB: Compressed tables use zlib 1.2.8
140808 18:35:02 [Note] InnoDB: Using Linux native AIO
140808 18:35:02 [Note] InnoDB: Using CPU crc32 instructions
140808 18:35:02 [Note] InnoDB: Initializing buffer pool, size = 8.0M
140808 18:35:02 [Note] InnoDB: Completed initialization of buffer pool
140808 18:35:02 [Note] InnoDB: Highest supported file format is Barracuda.
140808 18:35:02 [Note] InnoDB: 128 rollback segment(s) are active.
140808 18:35:02 [Note] InnoDB: Waiting for purge to start
140808 18:35:02 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 17630569340
140808 18:35:02 [Note] Plugin 'FEEDBACK' is disabled.
Fri Aug  8 18:35:03 2014 Tokudb recovery starting in env /var/lib/mysql/
Fri Aug  8 18:35:04 2014 Tokudb recovery scanning backward from 2822092
Fri Aug  8 18:35:12 2014 Tokudb recovery bw_end_checkpoint at 1524828 timestamp 1407507854498401 xid 1524629 (bw_newer)
Fri Aug  8 18:35:12 2014 Tokudb recovery bw_begin_checkpoint at 1524629 timestamp 1407507854497794 (bw_between)
Fri Aug  8 18:35:12 2014 Tokudb recovery turning around at begin checkpoint 1524629 time 607
Fri Aug  8 18:35:12 2014 Tokudb recovery starts scanning forward to 2822092 from 1524629 left 1297463 (fw_between)
Fri Aug  8 18:35:19 2014 Tokudb recovery scanning forward to 2822092 at 2030629 left 791463 (fw_newer)
Fri Aug  8 18:35:29 2014 Tokudb recovery closing 194 dictionaries
Fri Aug  8 18:35:31 2014 Tokudb recovery making a checkpoint
Fri Aug  8 18:35:31 2014 Tokudb recovery done
140808 18:35:31 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
140808 18:35:32 [ERROR] Error in Log_event::read_log_event(): 'Event too small', data_len: 0, event_type: 0
140808 18:35:32 [Note] Starting crash recovery...
140808 18:35:32 [Note] Crash recovery finished.
140808 18:35:32 [Note] Server socket created on IP: '0.0.0.0'.
140808 18:35:32 [Note] Event Scheduler: Loaded 0 events
140808 18:35:32 [Note] WSREP: Read nil XID from storage engines, skipping position init
140808 18:35:32 [Note] WSREP: wsrep_load(): loading provider library 'none'
140808 18:35:32 [Note] [Debug] WSREP: dummy_init
140808 18:35:32 [Note] mysqld: ready for connections.
Version: '10.0.12-MariaDB-1~trusty-wsrep-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution, wsrep_25.10.r4002

mysql

System start's and no entry's at bacula database but all tabel's are created.

MariaDB [(none)]> \s
--------------
mysql  Ver 15.1 Distrib 10.0.10-MariaDB, for Linux (x86_64) using readline 5.1

Connection id:		83
Current database:	
Current user:		root@localhost
SSL:			Not in use
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server:			MariaDB
Server version:		10.0.12-MariaDB-1~trusty-wsrep-log mariadb.org binary distribution, wsrep_25.10.r4002
Protocol version:	10
Connection:		Localhost via UNIX socket
Server characterset:	latin1
Db     characterset:	latin1
Client characterset:	utf8
Conn.  characterset:	utf8
UNIX socket:		/var/run/mysqld/mysqld.sock
Uptime:			5 min 6 sec

Threads: 1  Questions: 158  Slow queries: 0  Opens: 175  Flush tables: 1  Open tables: 88  Queries per second avg: 0.516
--------------

MariaDB [(none)]> 

MariaDB [bacula]> SELECT concat(TABLE_NAME,':',ENGINE) FROM Information_schema.TABLES  WHERE TABLE_SCHEMA='bacula';
+-------------------------------+
| concat(TABLE_NAME,':',ENGINE) |
+-------------------------------+
| BaseFiles:TokuDB              |
| CDImages:InnoDB               |
| Client:TokuDB                 |
| Counters:TokuDB               |
| Device:TokuDB                 |
| File:InnoDB                   |
| FileSet:TokuDB                |
| Filename:TokuDB               |
| Job:InnoDB                    |
| JobHisto:InnoDB               |
| JobMedia:InnoDB               |
| Location:InnoDB               |
| LocationLog:InnoDB            |
| Log:InnoDB                    |
| Media:InnoDB                  |
| MediaType:InnoDB              |
| Path:InnoDB                   |
| PathHierarchy:InnoDB          |
| PathVisibility:InnoDB         |
| Pool:InnoDB                   |
| RestoreObject:InnoDB          |
| Status:InnoDB                 |
| Storage:InnoDB                |
| UnsavedFiles:InnoDB           |
| Version:InnoDB                |
+-------------------------------+
25 rows in set (0.01 sec)

MariaDB [bacula]>

Show

Joni-Pekka Kurronen added a comment - 2014-08-08 18:22 - edited mysql < bacula.sql command run's abt. 8 hours ending TOTAL CRASH OF UBUNTU Trusty desktop whit 32GB ram, 1T RADI1 ( 2x1T ),... after it it take 3 hours to sync RAID1 disk. bacula.sql size is abt 3G and useing Tokudb Log say's very litle: /usr/sbin/mysqld, Version: 10.0.12-MariaDB-1~trusty-wsrep-log (mariadb.org binary distribution, wsrep_25.10.r4002). started with: Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock Time Id Command Argument This error can be reproduced but take's 8 hours. Manualy started to starup, need's HUGE SETTING before start. joni@mpi1:~$ sudo su root@mpi1:/home/joni# echo never >/sys/kernel/mm/transparent_hugepage/enabled root@mpi1:/home/joni# exit exit joni@mpi1:~$ sudo mysqld 140808 18:35:02 [Warning] option 'table_definition_cache': unsigned value 100 adjusted to 400 140808 18:35:02 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/local/mysql/share/english/ 140808 18:35:02 [Warning] Use --lc-messages-dir without language specific part instead. 140808 18:35:02 [ERROR] Error message file 'errmsg.sys' is probably from and older version of MariaDB / MYSQL as it doesn't contain all error messages 2014-08-08 18:35:02 7f61397067c0 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator. 140808 18:35:02 [Note] InnoDB: Using mutexes to ref count buffer pool pages 140808 18:35:02 [Note] InnoDB: The InnoDB memory heap is disabled 140808 18:35:02 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 140808 18:35:02 [Note] InnoDB: Compressed tables use zlib 1.2.8 140808 18:35:02 [Note] InnoDB: Using Linux native AIO 140808 18:35:02 [Note] InnoDB: Using CPU crc32 instructions 140808 18:35:02 [Note] InnoDB: Initializing buffer pool, size = 8.0M 140808 18:35:02 [Note] InnoDB: Completed initialization of buffer pool 140808 18:35:02 [Note] InnoDB: Highest supported file format is Barracuda. 140808 18:35:02 [Note] InnoDB: 128 rollback segment(s) are active. 140808 18:35:02 [Note] InnoDB: Waiting for purge to start 140808 18:35:02 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.17-65.0 started; log sequence number 17630569340 140808 18:35:02 [Note] Plugin 'FEEDBACK' is disabled. Fri Aug 8 18:35:03 2014 Tokudb recovery starting in env /var/lib/mysql/ Fri Aug 8 18:35:04 2014 Tokudb recovery scanning backward from 2822092 Fri Aug 8 18:35:12 2014 Tokudb recovery bw_end_checkpoint at 1524828 timestamp 1407507854498401 xid 1524629 (bw_newer) Fri Aug 8 18:35:12 2014 Tokudb recovery bw_begin_checkpoint at 1524629 timestamp 1407507854497794 (bw_between) Fri Aug 8 18:35:12 2014 Tokudb recovery turning around at begin checkpoint 1524629 time 607 Fri Aug 8 18:35:12 2014 Tokudb recovery starts scanning forward to 2822092 from 1524629 left 1297463 (fw_between) Fri Aug 8 18:35:19 2014 Tokudb recovery scanning forward to 2822092 at 2030629 left 791463 (fw_newer) Fri Aug 8 18:35:29 2014 Tokudb recovery closing 194 dictionaries Fri Aug 8 18:35:31 2014 Tokudb recovery making a checkpoint Fri Aug 8 18:35:31 2014 Tokudb recovery done 140808 18:35:31 [Note] Recovering after a crash using /var/log/mysql/mysql-bin 140808 18:35:32 [ERROR] Error in Log_event::read_log_event(): 'Event too small', data_len: 0, event_type: 0 140808 18:35:32 [Note] Starting crash recovery... 140808 18:35:32 [Note] Crash recovery finished. 140808 18:35:32 [Note] Server socket created on IP: '0.0.0.0'. 140808 18:35:32 [Note] Event Scheduler: Loaded 0 events 140808 18:35:32 [Note] WSREP: Read nil XID from storage engines, skipping position init 140808 18:35:32 [Note] WSREP: wsrep_load(): loading provider library 'none' 140808 18:35:32 [Note] [Debug] WSREP: dummy_init 140808 18:35:32 [Note] mysqld: ready for connections. Version: '10.0.12-MariaDB-1~trusty-wsrep-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution, wsrep_25.10.r4002 mysql System start's and no entry's at bacula database but all tabel's are created. MariaDB [(none)]> \s -------------- mysql Ver 15.1 Distrib 10.0.10-MariaDB, for Linux (x86_64) using readline 5.1 Connection id: 83 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server: MariaDB Server version: 10.0.12-MariaDB-1~trusty-wsrep-log mariadb.org binary distribution, wsrep_25.10.r4002 Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: latin1 Db characterset: latin1 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/run/mysqld/mysqld.sock Uptime: 5 min 6 sec Threads: 1 Questions: 158 Slow queries: 0 Opens: 175 Flush tables: 1 Open tables: 88 Queries per second avg: 0.516 -------------- MariaDB [(none)]> MariaDB [bacula]> SELECT concat(TABLE_NAME,':',ENGINE) FROM Information_schema.TABLES WHERE TABLE_SCHEMA='bacula'; +-------------------------------+ | concat(TABLE_NAME,':',ENGINE) | +-------------------------------+ | BaseFiles:TokuDB | | CDImages:InnoDB | | Client:TokuDB | | Counters:TokuDB | | Device:TokuDB | | File:InnoDB | | FileSet:TokuDB | | Filename:TokuDB | | Job:InnoDB | | JobHisto:InnoDB | | JobMedia:InnoDB | | Location:InnoDB | | LocationLog:InnoDB | | Log:InnoDB | | Media:InnoDB | | MediaType:InnoDB | | Path:InnoDB | | PathHierarchy:InnoDB | | PathVisibility:InnoDB | | Pool:InnoDB | | RestoreObject:InnoDB | | Status:InnoDB | | Storage:InnoDB | | UnsavedFiles:InnoDB | | Version:InnoDB | +-------------------------------+ 25 rows in set (0.01 sec) MariaDB [bacula]>

Hide

Permalink

Elena Stepanova added a comment - 2014-08-08 19:31

Hi,

exec "echo never >/sys/kernel/mm/transparent_hugepage/enabled"
That's needed to get TokuDB functionin and maraiadb goin up.

Yes, this is a documented TokuDB requirement which comes from the engine: https://mariadb.com/kb/en/mariadb/documentation/storage-engines/tokudb/how-to-enable-tokudb-in-mariadb/ .
And in your case the whole MariaDB server won't get started, because you set TokuDB as a defalut storage engine, and without the default engine server does not run.

mysqld server did hang and prevented normal start and stop,
no error log messages , THAT WAS BEFORE SCRIPT CHANGE

It's difficult to say anything about hanging without any error messages or a stack trace. As said above, the server was not expected to start, but it should not hang, rather just go down with the failure.

I COULD NOT TERMINATE started mysqld -v on terminal whit Ctrl-X-C !

Over many versions and many years by now, mysqld only handles SIGINT (Ctrl+C) when it's started with --gdb option:

mysqld now only adds an interrupt handler for the SIGINT signal if you start it with the new --gdb option. This is because some MySQL users encountered strange problems when they accidentally sent SIGINT to mysqld threads

http://dev.mysql.com/doc/refman/5.0/fr/news-4-0-14.html

Complains that error.log is old format ! , WHILE GETTING PROBLEM SORTED DUE NO ERROR LOG

I don't quite understand this part. Please copy-paste the exact message that you received.

After crash mysqld start's, but dose not allow connections, at this time i will wait two hours,...
Log say's very litle:

What you quoted looks like the general log. You need to find the error log. It might be a separate file or in your syslog or alike.

Show

Elena Stepanova added a comment - 2014-08-08 19:31 Hi, exec "echo never >/sys/kernel/mm/transparent_hugepage/enabled" That's needed to get TokuDB functionin and maraiadb goin up. Yes, this is a documented TokuDB requirement which comes from the engine: https://mariadb.com/kb/en/mariadb/documentation/storage-engines/tokudb/how-to-enable-tokudb-in-mariadb/ . And in your case the whole MariaDB server won't get started, because you set TokuDB as a defalut storage engine, and without the default engine server does not run. mysqld server did hang and prevented normal start and stop, no error log messages , THAT WAS BEFORE SCRIPT CHANGE It's difficult to say anything about hanging without any error messages or a stack trace. As said above, the server was not expected to start, but it should not hang, rather just go down with the failure. I COULD NOT TERMINATE started mysqld -v on terminal whit Ctrl-X-C ! Over many versions and many years by now, mysqld only handles SIGINT (Ctrl+C) when it's started with --gdb option: mysqld now only adds an interrupt handler for the SIGINT signal if you start it with the new --gdb option. This is because some MySQL users encountered strange problems when they accidentally sent SIGINT to mysqld threads http://dev.mysql.com/doc/refman/5.0/fr/news-4-0-14.html Complains that error.log is old format ! , WHILE GETTING PROBLEM SORTED DUE NO ERROR LOG I don't quite understand this part. Please copy-paste the exact message that you received. After crash mysqld start's, but dose not allow connections, at this time i will wait two hours,... Log say's very litle: What you quoted looks like the general log. You need to find the error log. It might be a separate file or in your syslog or alike.

Hide

Permalink

Elena Stepanova added a comment - 2014-08-08 19:35 - edited

140808 18:35:02 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/local/mysql/share/english/
140808 18:35:02 [Warning] Use --lc-messages-dir without language specific part instead.
140808 18:35:02 [ERROR] Error message file 'errmsg.sys' is probably from and older version of MariaDB / MYSQL as it doesn't contain all error messages

These two warnings (and quite possibly the error too) are caused by this line in your cnf:

language = /usr/local/mysql/share/english

Try to comment it and see if the server is able to find lc-messages-dir on its own.

140808 18:35:31 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
140808 18:35:32 [ERROR] Error in Log_event::read_log_event(): 'Event too small', data_len: 0, event_type: 0
140808 18:35:32 [Note] Starting crash recovery...

Maybe you really have corrupted binary logs?
Do you even need them?
Do you have replication (apart from Galera), or use them for backup?

Show

Elena Stepanova added a comment - 2014-08-08 19:35 - edited 140808 18:35:02 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /usr/local/mysql/share/english/ 140808 18:35:02 [Warning] Use --lc-messages-dir without language specific part instead. 140808 18:35:02 [ERROR] Error message file 'errmsg.sys' is probably from and older version of MariaDB / MYSQL as it doesn't contain all error messages These two warnings (and quite possibly the error too) are caused by this line in your cnf: language = /usr/local/mysql/share/english Try to comment it and see if the server is able to find lc-messages-dir on its own. 140808 18:35:31 [Note] Recovering after a crash using /var/log/mysql/mysql-bin 140808 18:35:32 [ERROR] Error in Log_event::read_log_event(): 'Event too small', data_len: 0, event_type: 0 140808 18:35:32 [Note] Starting crash recovery... Maybe you really have corrupted binary logs? Do you even need them? Do you have replication (apart from Galera), or use them for backup?

Hide

Permalink

Joni-Pekka Kurronen added a comment - 2014-08-09 10:43

hi,

Changes to my.cnf removed total crash.

So I assume you know reason for TOTAL CRASH and at any case
it is not acceptable.

Thank you Elena to workaround !

yours,
joni

Show

Joni-Pekka Kurronen added a comment - 2014-08-09 10:43 hi, Changes to my.cnf removed total crash. So I assume you know reason for TOTAL CRASH and at any case it is not acceptable. Thank you Elena to workaround ! yours, joni

Hide

Permalink

Elena Stepanova added a comment - 2014-08-09 10:58

Which TOTAL CRASH do you mean? Please specify, since you described several mutually unrelated issues, mainly caused by misconfiguration.
If there was a real crash and not just server going down gracefully due to a wrong configuration, please provide the error log and/or stack trace.

Show

Elena Stepanova added a comment - 2014-08-09 10:58 Which TOTAL CRASH do you mean? Please specify, since you described several mutually unrelated issues, mainly caused by misconfiguration. If there was a real crash and not just server going down gracefully due to a wrong configuration, please provide the error log and/or stack trace.

Hide

Permalink

Joni-Pekka Kurronen added a comment - 2014-08-28 02:32

REAL CRASH. TOTAL. power off 4 seconds and start again.

Show

Joni-Pekka Kurronen added a comment - 2014-08-28 02:32 REAL CRASH. TOTAL. power off 4 seconds and start again.

Hide

Permalink

Joni-Pekka Kurronen added a comment - 2014-08-28 02:33

Solution: stop useing tokudb !

Show

Joni-Pekka Kurronen added a comment - 2014-08-28 02:33 Solution: stop useing tokudb !

Hide

Permalink

Elena Stepanova added a comment - 2014-09-15 15:03

Closing as incomplete for now; if you have any new information (namely, explanation which TOTAL/REAL CRASH you are talking about, with the quote from the error log including the stack trace), please comment to re-open the issue.

Show

Elena Stepanova added a comment - 2014-09-15 15:03 Closing as incomplete for now; if you have any new information (namely, explanation which TOTAL/REAL CRASH you are talking about, with the quote from the error log including the stack trace), please comment to re-open the issue.

People

Assignee:

Unassigned

Reporter:

Joni-Pekka Kurronen

Votes:

0 Vote for this issue

Watchers:

3 Start watching this issue

Dates

Due:

2014-09-13

Created:

2014-08-08 10:43

Updated:

2014-09-15 15:03

Resolved:

2014-09-15 15:03

Agile

View on Board