CONNECT: Can not load JSON files over 12MB (roughly)

Description

CONNECT is not parsing JSON files over roughly 12MB. I receive the following error when I try to select from the table-

Error Code: 1296. Got error 174 'ParseString: Out of memory' from CONNECT 0.406 sec

I've tried every option available, and also tried increasing the amount of memory available. I am not sure if there is a way to increase the memory blocks available to 'Sarea' through either a patch or server variable?

I have tried each option, including-

_SET GLOBAL connect_work_size = 4199999999;
SET GLOBAL connect_use_tempfile=YES;
SET GLOBAL connect_json_grp_size=2147483647;_

I tried to use multiple files using a wildcard (this is what I really need - I have 2GB of files, smallest being under 1MB and largest being over 100MB in size.) My use case is around parsing some operations data from a custom application and just pull some counts. CONNECT seemed like it was fit for this.

My data is an array of objects. My table creation statement looks like the following (I tried everything, from the most simple autodetection to manual mapping, and also trying to force the row size to see if I could get it to allocate more memory - nothing works)-

CREATE TABLE `raw_sr` (
`userid` varchar(60) field_format='userid:*' DEFAULT NULL,
`fromgoogle_message_id` varchar(45) field_format='fromgoogle_message_id:*' DEFAULT NULL,
`fromswift_container_id` int field_format='fromswift_container_id:*' DEFAULT NULL,
`fromswift_object_id` int field_format='fromswift_object_id:*' DEFAULT NULL
)
ENGINE=CONNECT DEFAULT CHARSET=latin1 `TABLE_TYPE`='JSON'
MULTIPLE=1 MAPPED=1 HUGE=1 READONLY=1
`FILE_NAME`='ALL/*_SR.json'
option_list='Pretty=2,Jmode=1' lrecl=1024;

It failed loading multiple, it also fails loading a single large file. None of the options make a difference. Is CONNECT JSON designed for this use case? Is there a way to get it to allocate more memory for JSON parsing? I have 16GB, MySQL usage is around ~2GB. I don't care if it uses all the memory, I just want the data to be exposed in the database engine. Please advise. Thanks you.

Environment

Windows Server 2012 R2 (x64) with 16GB of memory

Assignee

Olivier Bertrand

Reporter

NicholasF

Labels

None

Components

Fix versions

Affects versions

Priority

Major
Configure