connect_work_size is a (big)integer variable. To my knowlege there is no provision in MariaDB to use not numeric characters to specify it as well in .cnf file or using the set command. Therefore, 512M is interpreted as 512 and because being too small is replaced by the default value.
I tried for instance to specify:

set @@global.aria_log_file_size=10M

and got the error message "Incorrect argument type for variable 'aria_log_file'"
A possible fix would be to specify these numeric variables as string and do the conversion. Do you think it is advisable?

Thanks for your comment Olivier.

When using the set command you have to use a integer variable, but that is not the case for .cnf files.
That is why, I am getting the below.

show variables like 'connect_work_size';
+-------------------+-----------+
| Variable_name     | Value     |
+-------------------+-----------+
| connect_work_size | 536870912 |
+-------------------+-----------+
1 row in set (0.00 sec)

Instead of "512M " I wrote in my connect.cnf

connect_work_size=536870912

and restarted MariaDB and tried the same command but I get the below error which indicates my connect_work_size is not set properly.
As I wrote in my original post, it works properly when I use the set command instead.

PlugSubAlloc: Not enough memory in Work area for request of 80 (used=67108800 fr
ee=64)

I am not familiar with using .cnf file with a plugin. Could you tell me where to place such a file so I can test and see what happens (on Windows)

I'm not familiar with the Windows environment.
I am using MariaDB for CentOS release 6.5 (X86_64) and all .cnf files to set parameters is located below
/etc/my.cnf.d/
You can't test unless you use a CentOS like environment.

I don't know exactly what happens on your system but here is what I did on Windows. To start the server I added the mysqld option:

--defaults-extra-file=connect.cnf

And I put in the base directory the file connect.cnf:

[mysqld]
connect_work_size=512M

On debug mode, I was able to follow what was done in getopt_ull, eval_num_suffix and getopt_ull_limit_value (in my_getopt.c)
All was normal and the value was set to 536870912 as expected.
Anyway, no CONNECT code is called in this process.