MySQL Cluster NDB Restore

Every now and again you can get a failure also with MySQL Cluster NDB and the only way to bring back online is with the restore command.


Let's look how , shall we ?

After the failure of either Data Nodes we look at the log files and following error we are going to find :

"Error while reading REDO log. from......"

Something put them out of use !

Following you'll find the commands to execute a complete restore process :


1) If other Nodes as Management or Sql Nodes are still online, bring them offline.

2) Bring online the Management Nodes : ndb_mgmd  --configdir=/home/mysql/mysqlCluster -f /home/mysql/mysqlCluster/config.ini

3) Bring online the Data Nodes :  ndbd --initial

Be careful !!  The option "--initial" is going to reset and destroy all information stored in the Data Nodes.
   
4) Leaving the Sql Nodes offline execute the following commands on both the data nodes:

On the first Data Nodes :

ndb_restore -r -m -b 1 -n 3 /home/mysql/mysqlClusterData/BACKUP/BACKUP-1

Backup Id = 1
Nodeid = 3
backup path = /home/mysql/mysqlClusterData/BACKUP/BACKUP-1
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1.3.ctl'
File size 21388 bytes
Backup version in files: ndb-6.3.11 ndb version: mysql-5.6.15 ndb-7.3.4
Stop GCP of Backup: 670480
Connected to ndb!!
Successfully restored table `test/def/ciccio`
Successfully restored table event REPL$test/ciccio
Create foreign keys
Create foreign keys done
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1-0.3.Data'
File size 1556696 bytes
_____________________________________________________
Processing data in table: mysql/def/NDB$BLOB_7_3(8) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_index_stat_sample(5) fragment 0
_____________________________________________________
Processing data in table: sys/def/NDB$EVENTS_0(3) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_apply_status(9) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_index_stat_head(4) fragment 0
_____________________________________________________
Processing data in table: test/def/ciccio(10) fragment 0
_____________________________________________________
Processing data in table: sys/def/SYSTAB_0(2) fragment 0
_____________________________________________________
Processing data in table: mysql/def/ndb_schema(7) fragment 0
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1.3.log'
File size 52 bytes
Restored 54663 tuples and 0 log entries
NDBT_ProgramExit: 0 - OK
   

On the second Data Node :

ndb_restore -r -b 1 -n 4 /home/mysql/mysqlClusterData/BACKUP/BACKUP-1

Backup Id = 1
Nodeid = 4
backup path = /home/mysql/mysqlClusterData/BACKUP/BACKUP-1
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1.4.ctl'
File size 21388 bytes
Backup version in files: ndb-6.3.11 ndb version: mysql-5.6.15 ndb-7.3.4
Stop GCP of Backup: 670480
Connected to ndb!!
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1-0.4.Data'
File size 1575096 bytes
_________________________________________________
Processing data in table: mysql/def/NDB$BLOB_7_3(8) fragment 1
_____________________________________________________
Processing data in table: mysql/def/ndb_index_stat_sample(5) fragment 1
_____________________________________________________
Processing data in table: sys/def/NDB$EVENTS_0(3) fragment 1
_____________________________________________________
Processing data in table: mysql/def/ndb_apply_status(9) fragment 1
_____________________________________________________
Processing data in table: mysql/def/ndb_index_stat_head(4) fragment 1
_____________________________________________________
Processing data in table: test/def/ciccio(10) fragment 1
_____________________________________________________
Processing data in table: sys/def/SYSTAB_0(2) fragment 1
_____________________________________________________
Processing data in table: mysql/def/ndb_schema(7) fragment 1
Opening file '/home/mysql/mysqlClusterData/BACKUP/BACKUP-1/BACKUP-1.4.log'
File size 52 bytes
Restored 55338 tuples and 0 log entries
NDBT_ProgramExit: 0 - OK


Following you'll find some handy stuff about the option of ndb_restore command:

-m = restore meta
-r  = restore records
-b  = backup ID
-n  = node ID

1) On the Second Data Nodes we have executed the ndb_restore command without the "-m" option, because the metadata information has already been restored with the first execution on the first Data Node.

2) Due to native table partitioning method, that is used to divide rows between the Data Nodes, we have to execute the ndb_restore command on both the Data Nodes.

NB: Who is writing the Post doesn't know your environment so it's necessary to try it out in a test cluster database before.

This procedure ends up bringing back online the Data and the last step that we have to do is only to restart both Sql Nodes with the following command: mysqld_safe &


Feel free to leave a comment or ask for further queries regarding the Post.

See you in my next Blog !

Byeeeee