Details
Technical task
Minor
Description
Galera installation tests on CentOS 5 in buildbot fail with an obscure "Lost connection to MySQL server during query" error on enabling wsrep_provider:
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-x86/builds/2248/steps/test/logs/stdio
http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos5-amd64/builds/1981/steps/test/logs/stdio
I did some digging and it turned out that the actual cause is strict SELinux settings which Galera cannot deal with (it's documented in Galera FAQ: http://www.codership.com/wiki/doku.php?id=faq#qnothing_works_damnit)
In my experiments, it was enough to switch the level from Enforcing which it is in the VM now, to Permissive at runtime using setenforce. Setting the level permanently and rebooting the machine also helped, of course.
If the level is set/kept high on purpose, please consider a conditional setting it to Permissive at runtime for Galera tests (if you do so, please make sure setenforce comes with the full path, it's not on the PATH). If there is no particular reason to have it Enforcing by default, maybe it's easier to reconfigure it permanently.
I was checking 64-bit build, but I suppose the reason of the failure on x86 is the same.
Gliffy Diagrams
Attachments
Issue Links
- relates to
-
MDEV-4344 Galera: Server crashes on setting wsrep_cluster_address with old version of libgcc
-
- Open
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hi Daniel,
It turns out that wsrep crashes when the server is run with an old version of libgcc. The VM (vm-centos5-i386-install.qcow2) has
libgcc.i386 4.1.2-44.el5
while currently available is
libgcc i386 4.1.2-54.el5
I'll file a bug report for Galera so Codership could check why it crashes, but meanwhile we might upgrade the library in the VM (or maybe not, I don't know what our VM upgrade policies are). If you think it makes sense to upgrade, please go ahead; if not, please close this issue as fixed, since the initial one with SELinux has been resolved anyway.
I'm fine with upgrading the library in the VM
If you do that, could you please backup a VM image with the old library, so I could re-test the fix if the bug is fixed?
(the bug was filed as MDEV-4344)
With the new version of Galera, SELinux breaks tests on RHEL5 (I've seen and checked failures on x86, but I suppose it's the same on amd64, it just currently lags behind).
The failure is a bit different, it hides inside the init script, and shows up in messages as follows:
SELinux is preventing the mysqld from using potentially mislabeled files (./tmp.wBeQpv4640). For complete SELinux messages. run sealert -l 3efbde02-94d1-4e88-88ce-88a8a5cd3987
Switching to Permissive fixes it, please do so for Galera tests.
I suppose if we don't want to duplicate VM images and don't want to change it for regular tests, we can switch it off dynamically for the test, adding a conditional step
sudo bash -c "echo 0 > /selinux/enforce"
or alike.
RHEL and CentOS builders are green now, closing as fixed.
Let me know if other VMs need the same thing done to them and/or if we need to do more to the CentOS VMs.