GBase 8a 节点替换
创始人
2024-02-29 23:31:26
0次
节点替换 包括替换gcware、替换gcluster、替换gnode ，新节点可以是freenode节点（替换后节点IP改变），也可以是全新未安装过gbase的节点（替换后节点IP不变），本次内容是使用全新节点替换gnode，其他情况 以后有时间再补充
原始集群 192.168.61.1 【8a-1】 (gcware + gcluster + gnode)
192.168.61.2 【8a-2】 (gnode) 待替换节点
新节点 192.168.61.3 【8a-3】
目录
原始状态
删除被替换节点的 feventlog
生成新的distribution
将被替换的集群节点（原 192.168.61.2）机器网线拔出（或者直接关机），并将待替换的新机器（192.168.61.3）IP改为192.168.61.2
执行节点替换
再次数据重分布，将数据分布到新节点
删除旧的分布方案
问题记录
原始状态
[gbase@8a-1 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE====================================
| GBASE GCWARE CLUSTER INFORMATION |
====================================
| NodeName | IpAddress | gcware |
------------------------------------
| gcware1 | 192.168.61.1 | OPEN |
------------------------------------
======================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
======================================================
| NodeName | IpAddress | gcluster | DataState |
------------------------------------------------------
| coordinator1 | 192.168.61.1 | OPEN | 0 |
------------------------------------------------------
=============================================================
| GBASE CLUSTER FREE DATA NODE INFORMATION |
=============================================================
| NodeName | IpAddress | gnode | syncserver | DataState |
-------------------------------------------------------------
| FreeNode1 | 192.168.61.2 | OPEN | OPEN | 0 |
-------------------------------------------------------------
| FreeNode2 | 192.168.61.1 | OPEN | OPEN | 0 |
-------------------------------------------------------------0 virtual cluster
1 coordinator node
2 free data node设置被替换节点状态为 unavailable
当节点状态置为unavaliable时 该节点不再接收gcluster下发的SQL
[gbase@8a-1 gcinstall]$ gcadmin setnodestate 192.168.61.2 unavailable
[gbase@8a-1 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL====================================
| GBASE GCWARE CLUSTER INFORMATION |
====================================
| NodeName | IpAddress | gcware |
------------------------------------
| gcware1 | 192.168.61.1 | OPEN |
------------------------------------
======================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
======================================================
| NodeName | IpAddress | gcluster | DataState |
------------------------------------------------------
| coordinator1 | 192.168.61.1 | OPEN | 0 |
------------------------------------------------------
===============================================================================================================
| GBASE DATA CLUSTER INFORMATION |
===============================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------------
| node1 | 192.168.61.2 | 1 | UNAVAILABLE | | |
---------------------------------------------------------------------------------------------------------------
| node2 | 192.168.61.1 | 1 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------------
删除被替换节点的 feventlog
当集群各节点状态不一致时会生成feventlog，这个feventlog也是判断集群各节点是否一致的标准，当我们将节点状态置为unavailable时 这个节点已经与其他节点不一致了，需要手工删除feventlog，虽然再后面替换节点repalce.py这个里面也会删除feventlog, 但是我们选择先手工删除，这样在节点替换过程中 再次删除feventlog的时候会很快，可以保证在节点替换过程中 不会因为feventlog太大 而耗费时间过长，只要feventlog一致，那么集群就可以正常运转，同时也保证了替换节点过程中，瞬间删除feventlog ，集群可以正常工作，同时 在节点替换过程中，如果没有先将feventlog删除的话，那么节点替换过程中也会自动删除，只不过这个时间可能会很长，但是在这个时间段，下发过来的sql不会拒收，而是将SQL置为等待状态，等节点替换完继续执行SQL，这个特性也说明了节点替换过程中，业务不受影响
[gbase@8a-1 gcinstall]$ gcadmin rmfeventlog 192.168.61.2生成新的distribution
由于192.168.61.2将被替换，所以要将192.168.61.2上面的数据先转移，转移的方法就是先生成新的分布方案，再按照新的分布方案进行数据重分布。比如原集群的分布方案是192.168.61.1和192.168.61.2两个节点，192.168.61.2将被替换，那么新的分布方案就不能包含192.168.61.2了
原始集群的分布方案（Distribution ID = 1）
[gbase@8a-1 gcinstall]$ gcadmin showdistribution nodeDistribution ID: 1 | State: new | Total segment num: 2============================================================================================
| nodes | 192.168.61.2 | 192.168.61.1 |
--------------------------------------------------------------------------------------------
| primary | 1 | 2 |
| segments | | |
--------------------------------------------------------------------------------------------
|duplicate | 2 | 1 |
|segments 1| | |
============================================================================================
复制原始的分布方案
[gbase@8a-1 gcinstall]$ gcadmin getdistribution 1 distribution_info_1.xml
[gbase@8a-1 gcinstall]$ cat distribution_info_1.xml
修改新的分布方案
[gbase@8a-1 gcinstall]$ vi distribution_info_1.xml
如果被替换节点是主分片，那么将ip由被替换的节点改为改为该分片上的备份分片的IP，当前分片的备份分片为192.168.61.1，所以主分片这里改为 192.168.61.1
因为上面主分片已经改为192.168.61.1，这里的备份分片可以直接删掉
如果被替换节点是备份分片，那么这几行直接删除
[gbase@8a-1 gcinstall]$ vi gcChangeInfo_dis1.xml生成新的分布方案
[gbase@8a-1 gcinstall]$ gcadmin distribution gcChangeInfo_dis1.xml查看新的分布方案
此时多了distribution ID = 2 ,这个分布方案里面不包含被替换节点
[gbase@8a-1 gcinstall]$ gcadmin showdistribution nodeDistribution ID: 2 | State: new | Total segment num: 2====================================================
| nodes | 192.168.61.1 |
----------------------------------------------------
| primary | 1 |
| segments | 2 |
====================================================Distribution ID: 1 | State: old | Total segment num: 2============================================================================================
| nodes | 192.168.61.2 | 192.168.61.1 |
--------------------------------------------------------------------------------------------
| primary | 1 | 2 |
| segments | | |
--------------------------------------------------------------------------------------------
|duplicate | 2 | 1 |
|segments 1| | |
============================================================================================
使用Distribution = 2 ,将数据重新分布但不给被替换节点分配数据
没有重分布时 数据的分布情况如下（数据全部使用data_distribution_id = 1）
gbase> select index_name,tbname,data_distribution_id,vc_id from gbase.table_distribution;
+-------------------------------+--------------------+----------------------+---------+
| index_name | tbname | data_distribution_id | vc_id |
+-------------------------------+--------------------+----------------------+---------+
| gclusterdb.rebalancing_status | rebalancing_status | 1 | vc00001 |
| gclusterdb.dual | dual | 1 | vc00001 |
| testdb.t1 | t1 | 1 | vc00001 |
+-------------------------------+--------------------+----------------------+---------+
3 rows in set (Elapsed: 00:00:00.00)重分布后 数据的分布情况如下（数据改为使用 data_distribution_id = 2）
gbase> initnodedatamap;
Query OK, 0 rows affected, 3 warnings (Elapsed: 00:00:00.11)gbase> rebalance instance to 2; 这里 to 2 表示使用Distribution = 2 的分布方案
Query OK, 1 row affected (Elapsed: 00:00:00.03)gbase> select index_name,status,percentage,priority,host,distribution_id from gclusterdb.rebalancing_status;
+------------+-----------+------------+----------+--------------+-----------------+
| index_name | status | percentage | priority | host | distribution_id |
+------------+-----------+------------+----------+--------------+-----------------+
| testdb.t1 | COMPLETED | 100 | 5 | 192.168.61.1 | 2 |
+------------+-----------+------------+----------+--------------+-----------------+
1 row in set (Elapsed: 00:00:00.01)将被替换的集群节点（原 192.168.61.2）机器网线拔出（或者直接关机），并将待替换的新机器（192.168.61.3）IP改为192.168.61.2
[root@8a-3 ~]# ip a
2: ens33: mtu 1500 qdisc fq_codel state UP group default qlen 1000link/ether 00:0c:29:c6:d3:8e brd ff:ff:ff:ff:ff:ffinet 192.168.61.2/24 brd 192.168.61.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::61d5:26ac:7834:3674/64 scope link noprefixroute
[root@8a-3 ~]# cd /opt/gbase/
[root@8a-3 gbase]# ll #新节点原始状态 当前目录为空
total 0
[root@8a-3 gbase]# ll #replace.py 完成之后 再次查看这个目录 发现已经安装了gbase
total 0
drwxrwxr-x 2 gbase gbase 6 Nov 28 23:58 192.168.61.2
自己在虚拟机测试时，这一步非常重要，如果忘记这步，那么替换前后集群是没有变化的，只有先将旧节点关机，新节点开机 才可以达到正常替换的效果，不然替换完了，结果一看 新节点192.168.61.3上什么都没有
执行节点替换
[gbase@8a-1 gcinstall]$ ./replace.py --host=192.168.61.2 --type=data --dbaUser=gbase --dbaUserPwd=gbase --generalDBUser=gbase --generalDBPwd=gbase20110531 --overwrite 这里的host表示被替换节点的IP再次检查分布情况（替换后的分布方案 Distribution = 3）
[gbase@8a-1 gcinstall]$ gcadmin showdistributionDistribution ID: 3 | State: new | Total segment num: 2Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 192.168.61.2 | 1 | 192.168.61.1 |
------------------------------------------------------------------------------------------------------------------------
| 192.168.61.1 | 2 | 192.168.61.2 |
========================================================================================================================Distribution ID: 2 | State: old | Total segment num: 2Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 192.168.61.1 | 1 | |
------------------------------------------------------------------------------------------------------------------------
| 192.168.61.1 | 2 | |
========================================================================================================================
再次数据重分布，将数据分布到新节点
[gbase@8a-1 gcinstall]$ gccli -uroot -p
Enter password: GBase client 9.5.3.27.14_patch.1b41b5c1. Copyright (c) 2004-2022, GBase. All Rights Reserved.gbase> rebalance instance to 3;
Query OK, 1 row affected (Elapsed: 00:00:00.02)gbase> select * from gclusterdb.rebalancing_status;
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
| index_name | db_name | table_name | tmptable | start_time | end_time | status | percentage | priority | host | distribution_id |
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
| test.t | test | t | | 2022-11-29 00:08:30.294000 | NULL | RUNNING | 0 | 5 | 192.168.61.1 | 3 |
+------------+---------+------------+----------+----------------------------+----------+---------+------------+----------+--------------+-----------------+
1 row in set (Elapsed: 00:00:00.00)删除旧的分布方案
[gbase@8a-1 gcinstall]$ gcadmin rmdistribution 2问题记录
删除旧的分布方案时无法删除，提示还在使用
[gbase@8a-1 gcinstall]$ gcadmin rmdistribution 2
cluster distribution ID [2]
it will be removed now
please ensure this is ok, input [Y,y] or [N,n]: y
select count(*) from gbase.nodedatamap where data_distribution_id=2 result is not 0
refreshnodedatamap drop 2 failed, gcluster command error: Can not drop nodedatamap 2. Some table are using it.
gcadmin remove distribution: check whether distribution [2] is using failed
gcadmin remove distribution [2] failedgbase> select index_name,tbname,data_distribution_id,vc_id from gbase.table_distribution;
+-------------------------------+--------------------+----------------------+---------+
| index_name | tbname | data_distribution_id | vc_id |
+-------------------------------+--------------------+----------------------+---------+
| gclusterdb.rebalancing_status | rebalancing_status | 3 | vc00001 |
| test.t | t | 2 | vc00001 |
| gclusterdb.dual | dual | 3 | vc00001 |
+-------------------------------+--------------------+----------------------+---------+
3 rows in set (Elapsed: 00:00:00.00)gbase> rebalance table test.t to 3;
ERROR 1707 (HY000): gcluster command error: target table has been rebalancing.查看重分布状态 发现重分布一直处于running状态
gbase> select index_name,status,percentage,priority,host,distribution_id from gclusterdb.rebalancing_status;
+------------+---------+------------+----------+--------------+-----------------+
| index_name | status | percentage | priority | host | distribution_id |
+------------+---------+------------+----------+--------------+-----------------+
| test.t | RUNNING | 0 | 1 | 192.168.61.1 | 3 |
+------------+---------+------------+----------+--------------+-----------------+判断是不是锁住了，查看锁信息，果然被锁了
[gbase@8a-1 gcinstall]$ gcadmin showlock+=====================================================================================+| GCLUSTER LOCK |+=====================================================================================++-----------------------------+------------+---------------+--------------+------+----+| Lock name | owner | content | create time |locked|type|+-----------------------------+------------+---------------+--------------+------+----+| gc-event-lock |192.168.61.1| global master |20221129002719| TRUE | E |+-----------------------------+------------+---------------+--------------+------+----+| vc00001.hashmap_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S |+-----------------------------+------------+---------------+--------------+------+----+| vc00001.test.db_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S |+-----------------------------+------------+---------------+--------------+------+----+| vc00001.test.t.meta_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S |+-----------------------------+------------+---------------+--------------+------+----+|vc00001.test.t.rebalance_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | E |+-----------------------------+------------+---------------+--------------+------+----+|vc00001.test.table_space_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S |+-----------------------------+------------+---------------+--------------+------+----+Total : 6找出当前被锁住的进程ID为3388
gbase> show processlist;
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 1780 | Waiting for next activation | NULL |
| 3388 | gbase | 192.168.61.1:36000 | NULL | Query | 667 | Move table's slice | REBALANCE /*+ synchronous,vcid */ TABLE "vc00001"."test"."t" TO 3 |
| 3391 | root | localhost | NULL | Query | 0 | NULL | show processlist |
+------+-----------------+--------------------+------+---------+------+-----------------------------+-------------------------------------------------------------------+
3 rows in set (Elapsed: 00:00:00.00)查看被锁进程的锁状态
[gbase@8a-1 gcinstall]$ gcadmin showlock | grep 3388| vc00001.hashmap_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S || vc00001.test.db_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S || vc00001.test.t.meta_lock |192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S ||vc00001.test.t.rebalance_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | E ||vc00001.test.table_space_lock|192.168.61.1|3388(LWP:41885)|20221129004555| TRUE | S |后面就是排查哪个进程持有了锁没有释放，然后杀掉就好
有时需要先删除datamap
refreshnodedatamap drop 2
GBase 8a扩容完成后refreshnodedatamap drop报错：Can not drop nodedatamap ,Some table are using it. – 老紫竹的家
下一篇：单源点最短路径（输出路径）
相关内容
热门资讯
AWSECS：访问外部网络时出...
如果您在AWS ECS中部署了应用程序，并且该应用程序需要访问外部网络，但是无法正常访问，可能是因为...
AWSElasticBeans...
在Dockerfile中手动配置nginx反向代理。例如，在Dockerfile中添加以下代码：FR...
银河麒麟V10SP1高级服务器...
银河麒麟高级服务器操作系统简介： 银河麒麟高级服务器操作系统V10是针对企业级关键业务...
北信源内网安全管理卸载
北信源内网安全管理是一款网络安全管理软件，主要用于保护内网安全。在日常使用过程中，卸载该软件是一种常...
AWR报告解读
WORKLOAD REPOSITORY PDB report (PDB snapshots) AW...
AWS管理控制台菜单和权限
要在AWS管理控制台中创建菜单和权限，您可以使用AWS Identity and Access Ma...
ToDesk 远程工具安装及...
目录 前言 ToDesk 优势 ToDesk 下载安装 ToDesk 功能展示 文件传输 设备链接 ...
群晖外网访问终极解决方法：IP...
写在前面的话 受够了群晖的quickconnet的小水管了，急需一个新的解决方法&#x...
不能访问光猫的的管理页面
光猫是现代家庭宽带网络的重要组成部分，它可以提供高速稳定的网络连接。但是，有时候我们会遇到不能访问光...
Azure构建流程（Power...
这可能是由于配置错误导致的问题。请检查构建流程任务中的“发布构建制品”步骤，确保正确配置了“Arti...