Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8735

galera rsync sst - tar of binlog to file before transfer waste of IO and space

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.0.21-galera
    • Fix Version/s: None
    • Component/s: Galera SST
    • Labels:
      None
    • Environment:
      rhel6

      Description

      started sst transfer.

      donor ran out of space due to the created wsrep_sst_binlog.tar

      rather than creating the tar can the rsync method just add the binlog index files to the filter?

      https://github.com/MariaDB/server/blob/10.0-galera/scripts/wsrep_sst_rsync.sh#L159

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            danblack Daniel Black added a comment -

            rsync has a --include-from= which could point directly at the index file perhaps.

            Show
            danblack Daniel Black added a comment - rsync has a --include-from= which could point directly at the index file perhaps.
            Hide
            danblack Daniel Black added a comment - - edited

            Work in progress. patch wasn't sufficient to copy binlogs (or need it on both ends of connection).

            --- /usr/bin/wsrep_sst_rsync.orig       2015-09-03 15:23:23.443343327 +0900
            +++ /usr/bin/wsrep_sst_rsync    2015-09-04 08:12:08.562332162 +0900
            @@ -79,9 +79,7 @@
             MAGIC_FILE="$WSREP_SST_OPT_DATA/rsync_sst_complete"
             rm -rf "$MAGIC_FILE"
            
            -BINLOG_TAR_FILE="$WSREP_SST_OPT_DATA/wsrep_sst_binlog.tar"
             BINLOG_N_FILES=1
            -rm -f "$BINLOG_TAR_FILE" || :
            
             if ! [ -z $WSREP_SST_OPT_BINLOG ]
             then
            @@ -114,7 +112,7 @@
            
             # New filter - exclude everything except dirs (schemas) and innodb files
             FILTER=(-f '- /lost+found' -f '- /.fseventsd' -f '- /.Trashes'
            -        -f '+ /wsrep_sst_binlog.tar' -f '+ /ib_lru_dump' -f '+ /ibdata*' -f '+ /*/' -f '- /*')
            +        -f '+ /ib_lru_dump' -f '+ /ibdata*' -f '+ /*/' -f '- /*')
            
             if [ "$WSREP_SST_OPT_ROLE" = "donor" ]
             then
            @@ -148,16 +146,10 @@
                         # Prepare binlog files
                         pushd $BINLOG_DIRNAME &> /dev/null
                         binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index)
            -            binlog_files=""
                         for ii in $binlog_files_full
                         do
            -                binlog_files="$binlog_files $(basename $ii)"
            +                FILTER+=( -f  '+ /'$(basename $ii) )
                         done
            -            if ! [ -z "$binlog_files" ]
            -            then
            -                wsrep_log_info "Preparing binlog files for transfer:"
            -                tar -cvf $BINLOG_TAR_FILE $binlog_files >&2
            -            fi
                         popd &> /dev/null
                     fi
            
            @@ -307,7 +299,6 @@
                         # Clean up old binlog files first
                         rm -f ${BINLOG_FILENAME}.*
                         wsrep_log_info "Extracting binlog files:"
            -            tar -xvf $BINLOG_TAR_FILE >&2
                         for ii in $(ls -1 ${BINLOG_FILENAME}.*)
                         do
                             echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index
            @@ -329,6 +320,4 @@
                 exit 22 # EINVAL
             fi
            
            -rm -f $BINLOG_TAR_FILE || :
            -
             exit 0
            
            Show
            danblack Daniel Black added a comment - - edited Work in progress. patch wasn't sufficient to copy binlogs (or need it on both ends of connection). --- /usr/bin/wsrep_sst_rsync.orig 2015-09-03 15:23:23.443343327 +0900 +++ /usr/bin/wsrep_sst_rsync 2015-09-04 08:12:08.562332162 +0900 @@ -79,9 +79,7 @@ MAGIC_FILE="$WSREP_SST_OPT_DATA/rsync_sst_complete" rm -rf "$MAGIC_FILE" -BINLOG_TAR_FILE="$WSREP_SST_OPT_DATA/wsrep_sst_binlog.tar" BINLOG_N_FILES=1 -rm -f "$BINLOG_TAR_FILE" || : if ! [ -z $WSREP_SST_OPT_BINLOG ] then @@ -114,7 +112,7 @@ # New filter - exclude everything except dirs (schemas) and innodb files FILTER=(-f '- /lost+found' -f '- /.fseventsd' -f '- /.Trashes' - -f '+ /wsrep_sst_binlog.tar' -f '+ /ib_lru_dump' -f '+ /ibdata*' -f '+ /*/' -f '- /*') + -f '+ /ib_lru_dump' -f '+ /ibdata*' -f '+ /*/' -f '- /*') if [ "$WSREP_SST_OPT_ROLE" = "donor" ] then @@ -148,16 +146,10 @@ # Prepare binlog files pushd $BINLOG_DIRNAME &> /dev/null binlog_files_full=$(tail -n $BINLOG_N_FILES ${BINLOG_FILENAME}.index) - binlog_files="" for ii in $binlog_files_full do - binlog_files="$binlog_files $(basename $ii)" + FILTER+=( -f '+ /'$(basename $ii) ) done - if ! [ -z "$binlog_files" ] - then - wsrep_log_info "Preparing binlog files for transfer:" - tar -cvf $BINLOG_TAR_FILE $binlog_files >&2 - fi popd &> /dev/null fi @@ -307,7 +299,6 @@ # Clean up old binlog files first rm -f ${BINLOG_FILENAME}.* wsrep_log_info "Extracting binlog files:" - tar -xvf $BINLOG_TAR_FILE >&2 for ii in $(ls -1 ${BINLOG_FILENAME}.*) do echo ${BINLOG_DIRNAME}/${ii} >> ${BINLOG_FILENAME}.index @@ -329,6 +320,4 @@ exit 22 # EINVAL fi -rm -f $BINLOG_TAR_FILE || : - exit 0

              People

              • Assignee:
                nirbhay_c Nirbhay Choubey
                Reporter:
                danblack Daniel Black
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: