Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6284

Merge downstream Debian/Ubuntu packaging into upstream MariaDB

    Details

      Description

      == Motivation ==

      Merge downstream .deb-packaging into upstream MariaDB to unify, streamline and ease future release packaging. This will also fix some packaging errors in current MariaDB.org deb-packages.

      == Patch description ==

      List of notable changes:

      • Completely new Debian packaging standard (3.0 quilt) including rules file, patch conventions and .files -> .install file listing conventions
      • The will no longer be a debian/dist subfolder and the packaging of Debian an Ubuntu will be unified (= Debian will ship with AppArmor files included)
      • Support for Debian releases before Wheezy (7) and Ubuntu releases before Precise (12.04) will be dropped

      The upstream MariaDB contents of debian/* will not be fully identical to the downstream Debian/Ubuntu one.

      List of intended differences between Debian master and MariaDB master regarding debian/ contents:

      • control file: maintainer, uploaders, VCS and browser links
      • omitted packages (not allowed in Debian): libmariadbclient18, libmysqlclient18, libmariadbclient-dev, mysql-common
      • omitted plugins (problematic in Debian): Handler socket, TokuDB, Mroonga, Cassandra
      • keep autobake.sh in mariadb.org repo

      To close this bug please:

      1) Merge selected parts of debian/* from https://anonscm.debian.org/gitweb/?p=pkg-mysql/mariadb-10.0.git to https://github.com/MariaDB/server via pull requests arriving from branch ok-debpkg at https://github.com/ottok/mariadb

      2) Backport form MariaDB development head to 10.1, 10.0 and maybe even 5.5 commits that fix individual issues.

      == Quality assurance ==

      The new debian/* contents represents the state of the art of Debian packaging. The mariadb-5.5 and mariadb-10.0 has passed all Debian quality assurance and landed in the official Debian repositories. More details at https://wiki.debian.org/Teams/MySQL/MariaDB

      These same packages have also gone downstream to Ubuntu, has passed Ubuntu QA and are included since the Ubuntu 14.04 release.

      Buildbot test for ok-debpkg branch: https://buildbot.askmonty.org/buildbot/grid?branch=ok-debpkg&category=main

      == Roll-out planning ==

      This merge should be safe to release with 10.1. All new installs and upgrades using the MariaDB.org repo or mixing it with official Debian or Ubuntu repository contents is expected to work.

      Below are the different scenarios how releasing the new packging at MariaDB.org will affect current installs/upgrades:

      A) If users currently have a MariaDB.org repo enabled, those packages will have a + in their version number. Debian package manager will always consider e.g. 10.1.6+maria as more recent than 10.1.6 from the native repositories. Therefore people having MariaDB.org repositories will always get primarly the packages from MariaDB.org repositories. It is very unlikely that a Debian repository would ever have a more recent release than in MariaDB.org (e.g. 10.1.7 would override 10.1.6+maria, but that is not likely to take place as MariaDB.org always releases first).

      B) The new packaging is designed and tested that it allows seamless upgrades from old deb packaging generation to new generation. Whether the new packaging is from MariaDB.org or distro repositories does not matter.

      C) Backwards works only partially, but is a rare case (e.g. current Ubuntu users trying to upgrade from new style debs 10.0 to MariaDB.org repo old style debs 10.0 will see issues, e.g. MDEV-5977). As soon as this merge is done and MariaDB.org publishes packages using the new packaging this corner case is fixed.

      D) If users remove the MariaDB.org from their install, they will automatically have their 10.1.6+maria updated to Debian/Ubuntu native 5.5.39 once it is released in Debian.

      E) Like previously, upgrading from MySQL 5.5 to MariaDB 5.5 works seamlessly and backwards also. Updating from MariaDB 5.5 to 10.0 works seamlessly but backwards will have issues (downgrade flag, InnoDB log size mismatch etc) the package manager will not solve automatically. If users dump and import data manually, and in between completely purge and install the packages, then users can always migrate to whatever version.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              tturenko Timofey Turenko added a comment -

              Ubuntu 14.04 packages (MariaDB 5.5.37) contain libmysqld., but both libmysqld.so and libmysqld.a are in 'libmariadbd-dev' package while in RPM (for example CentOS 6) libmysqld.a is in MariaDB-devel package and libmysqld.so* are in MariaDB-server package.
              As a result everything that is build under Ubuntu 14.04 depends on 'libmariadbd-dev' (on development package!)
              Are we going to merge this packaging ito MariaDB?

              Show
              tturenko Timofey Turenko added a comment - Ubuntu 14.04 packages (MariaDB 5.5.37) contain libmysqld. , but both libmysqld.so and libmysqld.a are in 'libmariadbd-dev' package while in RPM (for example CentOS 6) libmysqld.a is in MariaDB-devel package and libmysqld.so* are in MariaDB-server package. As a result everything that is build under Ubuntu 14.04 depends on 'libmariadbd-dev' (on development package!) Are we going to merge this packaging ito MariaDB?
              Hide
              otto Otto Kekäläinen added a comment -

              About "Fix Version: 10.0" - please note that the merge request is for the 5.5 series only. I'll do a separate merge request for 10.0 and open a new JIRA issue for that, as the contents in debian/* for 10.0 is different enough to justify a separate issue. But before I push the 10.0 branch I'd like to see 5.5 merged first so we can learn from possible issues that arise in 5.5 work and fix not repeat them for 10.0.

              Show
              otto Otto Kekäläinen added a comment - About "Fix Version: 10.0" - please note that the merge request is for the 5.5 series only. I'll do a separate merge request for 10.0 and open a new JIRA issue for that, as the contents in debian/* for 10.0 is different enough to justify a separate issue. But before I push the 10.0 branch I'd like to see 5.5 merged first so we can learn from possible issues that arise in 5.5 work and fix not repeat them for 10.0.
              Hide
              otto Otto Kekäläinen added a comment -

              Sergei Golubchik This hasn't seen much progress because I've been busy with constantly updating and improving the Debian/Ubuntu packaging. Also the turnaround for request/review has been slow. What if we would meet and have a 1-2 day sprint to get this merge reviewed and pushed in one go?

              What if we also changed the strategy, and not merge the latest packaging into 10.0 and 5.5 but rather only on 10.1 and keep downstream and upstream Debian packaging in sync from 10.1 onwards?

              Show
              otto Otto Kekäläinen added a comment - Sergei Golubchik This hasn't seen much progress because I've been busy with constantly updating and improving the Debian/Ubuntu packaging. Also the turnaround for request/review has been slow. What if we would meet and have a 1-2 day sprint to get this merge reviewed and pushed in one go? What if we also changed the strategy, and not merge the latest packaging into 10.0 and 5.5 but rather only on 10.1 and keep downstream and upstream Debian packaging in sync from 10.1 onwards?
              Hide
              otto Otto Kekäläinen added a comment -
              Show
              otto Otto Kekäläinen added a comment - Sergei Golubchik Note that current https://github.com/ottok/mariadb-10.0/compare/upstream...master.patch is 45488 lines long..
              Hide
              serg Sergei Golubchik added a comment -

              Attached a complete review of the above: ottok.diff

              Show
              serg Sergei Golubchik added a comment - Attached a complete review of the above: ottok.diff
              Hide
              otto Otto Kekäläinen added a comment -

              Recent changes done to downstream packaging based on Serg's review:

              commit ada2c8c84271283eb0f604820d5994c9a057784e
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 22:40:16 2015 +0300
              
                  Recover mysql-upgrade dir/link handlig wrongly removed in f7caa041db
                  
                  The preinst is supposed to save possible mysql datadir link
                  and the postinst is supposed to recover it.
                  
                  Also unify namig of the variables that reference the data
                  and log directories.
              
              commit d190985e7ba7621c99f877a9451943c6e2ae5d54
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 22:02:54 2015 +0300
              
                  mysqld config: rename key_buffer -> key_buffer_size
                  
                  Short name version is deprecated. Use full name instead.
                  https://mariadb.com/kb/en/mariadb/myisam-system-variables/#key_buffer_size
              
              commit d8fb0f21e7e835336b37d9d3533d7276fe7a99b3
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 21:12:18 2015 +0300
              
                  Clean up old cruft from rules file after review by Sergei Golubchik
                  
                   * Don't set or use MYSQL_BUILD_CC as they were hard-coded anyway
                   * Remove unnecessary comment about how make translates -j option to --jobserver
                   * Assembler functions are long gone and references to them should be removed
                   * C and C++ should have same optimization levels, that is -O3
                   * BIG-JOINS are not used anywhere, remove them
                   * Make compilation comment nicer (visible via "show variables like 'version%';")
                   * Make -DDEB contain actual distro name, might be of later use and anyway better
                     than just simple '1' (true) value
                   * Not needed to have TAOCRYPT_OPT in CFLAGS, taocrypt is C++
                   * Libwrap is disabled by default
                   * INFO_BIN and INFO_SRC files are not used anymore, no need to install them.
                   * Use INSTALL_MYSQLTESTDIR instead of manual mv line
              
              commit a9bbb0c5dac55aefcad7131dc29a8b2a809cc3ed
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 18:23:25 2015 +0300
              
                  Unified config file layout with upstream .cnf layout
                  
                  Now the packages mariadb-common, mariadb-client and mariadb-server each
                  install their own cnf file. A system will not have extra cnf files around.
                  
                  Also the .cnf files list all possible stanzas that can be used to
                  compartmentalize settings for different binaries.
                  
                  Role model was upstream
                  https://github.com/MariaDB/server/tree/83ba48b7c670f6dba465325cafd808c91f551544/support-files/rpm
              
              commit 59c96c74e098767b2e01c0fa6b2e70b8bce592cc
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 18:00:27 2015 +0300
              
                  Simplyfy overkill utf8mb4 settings
                  
                  From https://dev.mysql.com/doc/refman/5.6/en/charset-connection.html
                  
                    A SET NAMES 'charset_name' statement is equivalent to these three statements:
                  
                    SET character_set_client = charset_name;
                    SET character_set_results = charset_name;
                    SET character_set_connection = charset_name;
                  
                  Therefore it is probably overkill to have 'SET NAMES'.
                  
                  Also using both - and _ in variable names in useless as they translate
                  to the same variable.
              
              commit 2f71f6834d93d9591cc187d6cce6e3dd7268bd33
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 17:46:24 2015 +0300
              
                  Update upstream info in d/copyright
              
              commit df02e679e29dabd1926f477c70d984cc168fc159
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 17:27:29 2015 +0300
              
                  The test suite must depend on the exact MariaDB version as tests are version specific.
                  
                  Depending on latest or any newer version is wrong, as tests might not be compatible
                  with some never program versions.
              
              commit e3fe9539f010270aac5aa6e70a2c0e561c749df0
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 17:16:11 2015 +0300
              
                  Wrap and sort
              
              commit ed276550168e9933f9e804bcf5f10da0cf4fe648
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 16:24:30 2015 +0300
              
                  Created package libmariadbd and ship libmysqld.so.18 in it.
              
              commit 0ed651b0cd8bafed1d43be339d499cc0214e54e5
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 15:39:47 2015 +0300
              
                  Add libdbd-mysql-perl as Depends instead of Suggests as libdbi-perl needs it.
                  
                  The package libdbi-perl is a dependency and it is useless unless libdbd-mysql-perl
                  is also installed. Thus both should be in Depends. These lines are now also
                  identical to those in the mysql-5.6 control file.
              
              commit be79bfaf31b9839205cb34f6bb1244c2f9b36ef5
              Author: Otto Kekäläinen <otto@seravo.fi>
              Date:   Tue Jul 21 08:39:45 2015 +0300
              
                  Updated d/changelog
              
              

              (full log at https://github.com/ottok/mariadb-10.0/commits/master)

              Preliminary branch for upstream merge is at https://github.com/MariaDB/server/compare/10.1...ottok:ok-debpkg

              Show
              otto Otto Kekäläinen added a comment - Recent changes done to downstream packaging based on Serg's review: commit ada2c8c84271283eb0f604820d5994c9a057784e Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 22:40:16 2015 +0300 Recover mysql-upgrade dir/link handlig wrongly removed in f7caa041db The preinst is supposed to save possible mysql datadir link and the postinst is supposed to recover it. Also unify namig of the variables that reference the data and log directories. commit d190985e7ba7621c99f877a9451943c6e2ae5d54 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 22:02:54 2015 +0300 mysqld config: rename key_buffer -> key_buffer_size Short name version is deprecated. Use full name instead. https://mariadb.com/kb/en/mariadb/myisam-system-variables/#key_buffer_size commit d8fb0f21e7e835336b37d9d3533d7276fe7a99b3 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 21:12:18 2015 +0300 Clean up old cruft from rules file after review by Sergei Golubchik * Don't set or use MYSQL_BUILD_CC as they were hard-coded anyway * Remove unnecessary comment about how make translates -j option to --jobserver * Assembler functions are long gone and references to them should be removed * C and C++ should have same optimization levels, that is -O3 * BIG-JOINS are not used anywhere, remove them * Make compilation comment nicer (visible via "show variables like 'version%';") * Make -DDEB contain actual distro name, might be of later use and anyway better than just simple '1' (true) value * Not needed to have TAOCRYPT_OPT in CFLAGS, taocrypt is C++ * Libwrap is disabled by default * INFO_BIN and INFO_SRC files are not used anymore, no need to install them. * Use INSTALL_MYSQLTESTDIR instead of manual mv line commit a9bbb0c5dac55aefcad7131dc29a8b2a809cc3ed Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 18:23:25 2015 +0300 Unified config file layout with upstream .cnf layout Now the packages mariadb-common, mariadb-client and mariadb-server each install their own cnf file. A system will not have extra cnf files around. Also the .cnf files list all possible stanzas that can be used to compartmentalize settings for different binaries. Role model was upstream https://github.com/MariaDB/server/tree/83ba48b7c670f6dba465325cafd808c91f551544/support-files/rpm commit 59c96c74e098767b2e01c0fa6b2e70b8bce592cc Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 18:00:27 2015 +0300 Simplyfy overkill utf8mb4 settings From https://dev.mysql.com/doc/refman/5.6/en/charset-connection.html A SET NAMES 'charset_name' statement is equivalent to these three statements: SET character_set_client = charset_name; SET character_set_results = charset_name; SET character_set_connection = charset_name; Therefore it is probably overkill to have 'SET NAMES'. Also using both - and _ in variable names in useless as they translate to the same variable. commit 2f71f6834d93d9591cc187d6cce6e3dd7268bd33 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 17:46:24 2015 +0300 Update upstream info in d/copyright commit df02e679e29dabd1926f477c70d984cc168fc159 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 17:27:29 2015 +0300 The test suite must depend on the exact MariaDB version as tests are version specific. Depending on latest or any newer version is wrong, as tests might not be compatible with some never program versions. commit e3fe9539f010270aac5aa6e70a2c0e561c749df0 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 17:16:11 2015 +0300 Wrap and sort commit ed276550168e9933f9e804bcf5f10da0cf4fe648 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 16:24:30 2015 +0300 Created package libmariadbd and ship libmysqld.so.18 in it. commit 0ed651b0cd8bafed1d43be339d499cc0214e54e5 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 15:39:47 2015 +0300 Add libdbd-mysql-perl as Depends instead of Suggests as libdbi-perl needs it. The package libdbi-perl is a dependency and it is useless unless libdbd-mysql-perl is also installed. Thus both should be in Depends. These lines are now also identical to those in the mysql-5.6 control file. commit be79bfaf31b9839205cb34f6bb1244c2f9b36ef5 Author: Otto Kekäläinen <otto@seravo.fi> Date: Tue Jul 21 08:39:45 2015 +0300 Updated d/changelog (full log at https://github.com/ottok/mariadb-10.0/commits/master ) Preliminary branch for upstream merge is at https://github.com/MariaDB/server/compare/10.1...ottok:ok-debpkg
              Hide
              otto Otto Kekäläinen added a comment - - edited

              Sergei Golubchik Pull request filed at https://github.com/MariaDB/server/pull/97

              It fixes at least MDEV-8667 and MDEV-6326 right now, and the new control file will prevent lots of future problems.

              As the control file logic is now almost in sync with that of the official Debian version, this pull request will prevent problems people might have who try to upgrade from 10.0 in Debian/Ubuntu to 10.1 from mariadb.org repos.

              Once this PR is accepted, I'll continue on the merge for other parts.

              Show
              otto Otto Kekäläinen added a comment - - edited Sergei Golubchik Pull request filed at https://github.com/MariaDB/server/pull/97 It fixes at least MDEV-8667 and MDEV-6326 right now, and the new control file will prevent lots of future problems. As the control file logic is now almost in sync with that of the official Debian version, this pull request will prevent problems people might have who try to upgrade from 10.0 in Debian/Ubuntu to 10.1 from mariadb.org repos. Once this PR is accepted, I'll continue on the merge for other parts.
              Hide
              serg Sergei Golubchik added a comment -

              merged

              Show
              serg Sergei Golubchik added a comment - merged
              Hide
              otto Otto Kekäläinen added a comment -

              Sergei GolubchikRasmus Johansson I think I should continue to do another round of backporting fixes to 10.1 to close multiple issues linked above so that 10.1 would ship with less broken packaging. However the current 10.1 master head fails in multiple tests and I don't have a clean starting point: https://buildbot.askmonty.org/buildbot/grid?branch=ok-debpkg-10.1&category=main

              None of those fails are due to Debian packaging. I'll wait until 10.1 master is all green before I start working on my branch.

              Show
              otto Otto Kekäläinen added a comment - Sergei Golubchik Rasmus Johansson I think I should continue to do another round of backporting fixes to 10.1 to close multiple issues linked above so that 10.1 would ship with less broken packaging. However the current 10.1 master head fails in multiple tests and I don't have a clean starting point: https://buildbot.askmonty.org/buildbot/grid?branch=ok-debpkg-10.1&category=main None of those fails are due to Debian packaging. I'll wait until 10.1 master is all green before I start working on my branch.

                People

                • Assignee:
                  otto Otto Kekäläinen
                  Reporter:
                  otto Otto Kekäläinen
                • Votes:
                  4 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 1 day, 4 hours, 20 minutes
                    1d 4h 20m