Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
Description
It looks like CassandraSE has very small timeouts. Any operation that takes longer than a few seconds ends like this:
MariaDB [dbt3]> delete from customer ;
ERROR 1928 (HY000): Internal error: 'TimedOutException: Default TException.'
MariaDB [dbt3]> delete from lineitem ;
ERROR 1928 (HY000): Internal error: 'TimedOutException: Default TException.'
MariaDB [dbt3]> delete from nation ;
ERROR 1928 (HY000): Internal error: 'TimedOutException: Default TException.'
MariaDB [dbt3]> delete from orders ;
I added calls to
s->setConnTimeout(1000*1000);
s->setRecvTimeout(1000*1000);
s->setSendTimeout(1000*1000);
on a created TSocket object s, but this didn't seem to help.
This needs to be investigated.
Gliffy Diagrams
Attachments
Issue Links
- is part of
-
MDEV-431 Cassandra storage engine
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
$8 = (org::apache::cassandra::TimedOutException *) 0x7f19a43ea668
Interesting thing: this is not an exception from communication between mysqld process and cassandra. This exception is defined in cassandra.thrift file:
/** RPC timeout was exceeded. either a node failed mid-operation, or load was too high, or the requested op was too large. */
exception TimedOutException {
}
and it was passed to us from inside Cassandra!
Cassandra by its design may return timeouts, and in that case the user is expected to retry the operation. We could have better retry logic in Cassandra SE, if there was enough interest in making use of it.
Dunno if this related, but asynchronous client also experiences timeout failures. They seem to happen at random, examples:
on_batch_mutate_fail obj 1 reason TimedOutException: Default TException. at 1181256 rows
on_batch_mutate_fail obj 2 reason TimedOutException: Default TException. at 392616 rows
on_batch_mutate_fail obj 1 reason TimedOutException: Default TException. at 343295 rows (9 reactor loops)
Need to find that timeout setting. And may be even add reconnect/retry functionality.