国产免费观看青青草原网站_久久精品视频色悠悠_影音先锋激情5566_日本一區二區三區免費高清在線_麻豆精品一区综合av_丰满的大屁股一区二区_男女啪啪免费网站_草莓视频app在线观看下载_午夜寂寞少妇AA片_混乱的生物课月老师后续视频

 
您現(xiàn)在的位置:首頁 ? 知識庫 ? Oracle Oracle
Oracle 12.2新特性 | 基于權(quán)重的節(jié)點驅(qū)逐
發(fā)布日期:2018-01-27

Oracle 12.2新特性 | 基于權(quán)重的節(jié)點驅(qū)逐

 2018-01-26 羅雪原 

 

前言

在Oracle Clusterware需要從集群中逐出特定節(jié)點或一組節(jié)點的情況時,基于服務器權(quán)重的節(jié)點驅(qū)逐可充當斷裂機制,通過服務器上的負載附加信息來識別要驅(qū)逐的節(jié)點或節(jié)點組。其中所有節(jié)點被驅(qū)逐的可能性是一樣的。


本課程視頻,點擊閱讀原文即可下載

基于權(quán)重的集群驅(qū)逐介紹

基于權(quán)重的集群驅(qū)逐(Server Weight-Based Node Eviction)是一項Oracle 12.2版本引入的一項新特性。在此之前,Oracle集群在處理腦裂問題時,通過判斷子群(各自獨立)的規(guī)模情況,來決定在腦裂問題發(fā)生時,終止哪個子群,判斷的依據(jù)如下:


  • 如果子群規(guī)模(包含節(jié)點數(shù)量)不等,則大的子群獲勝,例如,{1}  {2,3,4}后者獲勝,子群{1}被驅(qū)逐;

  • 如果子群規(guī)模(包含節(jié)點數(shù)量)相等,則包含最小節(jié)點號的子群獲勝,例如,{1,4}{2,3}, 子群{1,4}獲勝,子群{2,3}被驅(qū)逐。

 

而在Oracle 12.2版本上引入的這項新特性,一定程度上增加了我們對于集群的控制。這種控制使得我們能避免由于一些規(guī)則原來的限定,而減少對大范圍的應用產(chǎn)生影響。

 

注意:


以下針對weight以及權(quán)重表述的是同一個意思。

特性使用>>>>

使用前提條件


  • 權(quán)重(Weight)分配只能在admin-managed的節(jié)點生效

  • 可以對于server或者對于注冊到集群的應用進行權(quán)重(Weight)分配


>>>>

使用說明


  • 權(quán)重分配給server

使用crsctl set server css_critical yes命令

  • 權(quán)重分配給數(shù)據(jù)庫實例或服務

在srvctl add/modify databse 或 srvctladd/modify service命令中帶上“-css_critical yes”參數(shù)

  • 權(quán)重分配給非ora.*資源

在crsctl add/modify resource命令中,加入-attr "CSS_CRITICAL=yes"參數(shù)


注意:


有些權(quán)重的分配需要重啟集群才能生效,而有些資源的分配,則不需要重啟資源。


根據(jù)目前發(fā)現(xiàn),非ora.*資源并非所有都能直接加入屬性直接修改,但是屬性中可以看到有CSS_CRITICAL屬性,可能是當前版本未開發(fā)修改接口。

特性測試>>>>

實驗環(huán)境說明


  • 使用軟件環(huán)境說明:

OS:MacOS 10.11.6

VirtualBox:v5.1.30 r118389 (Qt5.6.3)

 

  • 虛擬機環(huán)境說明:

OEL(OracleEnterprise Linux) 6.5 , x86_64

Oracle 12.2.0.1 (2-node RAC)


>>>>

實驗前的準備工作


VirtualBox虛擬機中的網(wǎng)卡情況如下:



其中網(wǎng)卡1為Public網(wǎng)絡,網(wǎng)卡2為心跳網(wǎng)絡分別對應OEL6.5中的eth0網(wǎng)卡和eth1網(wǎng)卡。



為了模擬心跳網(wǎng)絡中斷,我們在系統(tǒng)中編寫了如下兩個腳本去模擬心跳網(wǎng)絡中斷以及恢復:


Milo-Mac:lab milo$ ls -l

total 16

-rwxr--r--  1 milo  staff  111  1 24 21:38 interconnect_down.sh

-rwxr--r--  1 milo  staff  109  1 24 21:39 interconnect_up.sh

Milo-Mac:lab milo$ 


#### 模擬心跳網(wǎng)絡中斷:

Milo-Mac:lab milo$ sh interconnect_down.sh 


查看虛擬機的網(wǎng)卡的連通情況:


[root@rac122a ~]# mii-tool eth0

eth0: no autonegotiation, 100baseTx-FD, link ok

[root@rac122a ~]# mii-tool eth1

eth1: autonegotiation restarted, no link


[root@rac122b ~]# mii-tool eth0

eth0: no autonegotiation, 100baseTx-FD, link ok

[root@rac122b ~]# mii-tool eth1

eth1: autonegotiation restarted, no link


可以看到在腳本運行后,私網(wǎng)網(wǎng)卡eth1顯示為no link表示網(wǎng)線沒有連接到網(wǎng)卡中,即我們認為心跳網(wǎng)絡出現(xiàn)故障。


Milo-Mac:lab milo$ ls -l

total 16

-rwxr--r--  1 milo  staff  111  1 24 21:38 interconnect_down.sh

-rwxr--r--  1 milo  staff  109  1 24 21:39 interconnect_up.sh

Milo-Mac:lab milo$ 


#### 恢復心跳網(wǎng)絡:

Milo-Mac:lab milo$ sh interconnect_up.sh 


查看虛擬機的網(wǎng)卡的連通情況:


[root@rac122a ~]# mii-tool eth0

eth0: no autonegotiation, 100baseTx-FD, link ok

[root@rac122a ~]# mii-tool eth1

eth1: no autonegotiation, 100baseTx-FD, link ok


[root@rac122b ~]# mii-tool eth0

eth0: no autonegotiation, 100baseTx-FD, link ok

[root@rac122b ~]# mii-tool eth1

eth1: no autonegotiation, 100baseTx-FD, link ok


可以看到在腳本運行后,私網(wǎng)網(wǎng)卡eth1顯示為link ok表示網(wǎng)線連接到網(wǎng)卡中,即我們認為心跳網(wǎng)絡恢復,當然實際情況是我們還需要重新禁用和啟用eth1網(wǎng)卡,才能使其完全恢復,因此需要在兩個節(jié)點都執(zhí)行以下腳本:

 

[root@rac122a ~]# sh recover_interconnect.sh

Device state: 3 (disconnected)

Active connection state: activated

Active connection path: /org/freedesktop/NetworkManager/ActiveConnection/2


上述相關腳本如下:

 

Milo-Mac:lab milo$ cat interconnect_down.sh 

VBoxManage controlvm "12c_rac_node1" setlinkstate2 off

VBoxManage controlvm "12c_rac_node2" setlinkstate2 off


Milo-Mac:lab milo$ cat interconnect_up.sh 

VBoxManage controlvm "12c_rac_node1" setlinkstate2 on

VBoxManage controlvm "12c_rac_node2" setlinkstate2 on


[root@rac122a ~]# cat recover_interconnect.sh

ifdown eth1 && ifup eth1

經(jīng)過上述測試,我們已經(jīng)可以模擬心跳網(wǎng)絡故障以及恢復心跳網(wǎng)絡故障。

以下測試場景不再贅述,上述故障模擬以及故障恢復過程。


 

>>>>

測試場景


未設置任何權(quán)重的測試結(jié)果


 

[oracle@rac122a ~]$ crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is no.


[oracle@rac122b ~]$ crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is no.


 當前系統(tǒng)上存在的服務也為設置:

 

[oracle@rac122b ~]$ crsctl stat res ora.milodb.milodb_srv1.svc -f | grep CSS

CSS_CRITICAL=no

[oracle@rac122b ~]$ crsctl stat res ora.milodb.milodb_srv2.svc -f | grep CSS

CSS_CRITICAL=no


模擬心跳網(wǎng)絡故障

 

Milo-Mac:lab milo$ sh interconnect_down.sh


節(jié)點1集群的alert日志:


2018-01-25 08:17:33.423 [OCSSD(3896)]CRS-1612: Network communication with node rac122b (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.610 seconds

2018-01-25 08:17:41.435 [OCSSD(3896)]CRS-1611: Network communication with node rac122b (2) missing for 75% of timeout interval.  Removal of this node from cluster in 6.600 seconds

2018-01-25 08:17:45.438 [OCSSD(3896)]CRS-1610: Network communication with node rac122b (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.600 seconds


2018-01-25 08:17:49.554 [OCSSD(3896)]CRS-1607: Node rac122b is being evicted  in cluster incarnation 412297497; details at (:CSSNM00007:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc.


節(jié)點2集群的alert日志:


2018-01-25 08:17:34.128 [OCSSD(25756)]CRS-1612: Network communication with node rac122a (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.450 seconds

2018-01-25 08:17:41.140 [OCSSD(25756)]CRS-1611: Network communication with node rac122a (1) missing for 75% of timeout interval.  Removal of this node from cluster in 7.440 seconds

2018-01-25 08:17:46.193 [OCSSD(25756)]CRS-1610: Network communication with node rac122a (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.380 seconds

…省略部分信息


2018-01-25 08:17:50.820 [OCSSD(25756)]CRS-1608: This node was evicted by node 1, rac122a; details at (:CSSNM00005:) in /u01/app/grid/diag/crs/rac122b/crs/trace/ocssd.trc.


從上述信息來看,根據(jù)之前的集群腦裂出現(xiàn)的規(guī)則,在同等規(guī)模的子群中,節(jié)點1節(jié)點號小,因而存活,節(jié)點2被驅(qū)逐出集群。


 

設置節(jié)點2上的sever級別的權(quán)重


將心跳網(wǎng)絡恢復以及集群狀態(tài)恢復正常后,我們將對server級別的權(quán)重進行設置。


 


[root@rac122a ~]# crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is no.


[root@rac122b ~]# crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is no.


[root@rac122b ~]# crsctl set server css_critical yes

CRS-4416: Server attribute 'CSS_CRITICAL' successfully changed. Restart Oracle High Availability Services for new value to take effect. 


設置完成后,提示需要重啟OHAS才能生效。


$ srvctl stop instance -d milodb -i milodb2

# crsctl stop crs

# crsctl start crs


模擬網(wǎng)路故障:

 

Milo-Mac:lab milo$ sh interconnect_down.sh


此時,我們來觀察集群的一些日志情況:

 

節(jié)點1集群的alert日志:


2018-01-25 09:44:03.671 [OCSSD(3717)]CRS-1612: Network communication with node rac122b (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.890 seconds

2018-01-25 09:44:11.731 [OCSSD(3717)]CRS-1611: Network communication with node rac122b (2) missing for 75% of timeout interval.  Removal of this node from cluster in 6.830 seconds

2018-01-25 09:44:15.739 [OCSSD(3717)]CRS-1610: Network communication with node rac122b (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.830 seconds

2018-01-25 09:44:18.573 [OCSSD(3717)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc.

2018-01-25 09:44:18.573 [OCSSD(3717)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc

2018-01-25 09:44:18.616 [OCSSD(3717)]CRS-1652: Starting clean up of CRSD resources.


2018-01-25 09:44:20.608 [OCSSD(3717)]CRS-1608: This node was evicted by node 2, rac122b; details at (:CSSNM00005:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc.


節(jié)點2集群的alert日志:


2018-01-25 09:44:04.327 [OCSSD(8586)]CRS-1612: Network communication with node rac122a (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.020 seconds

2018-01-25 09:44:11.609 [OCSSD(8586)]CRS-1611: Network communication with node rac122a (1) missing for 75% of timeout interval.  Removal of this node from cluster in 6.730 seconds

2018-01-25 09:44:15.611 [OCSSD(8586)]CRS-1610: Network communication with node rac122a (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.730 seconds


2018-01-25 09:44:19.777 [OCSSD(8586)]CRS-1607: Node rac122a is being evicted in cluster incarnation 412336033; details at (:CSSNM00007:) in /u01/app/grid/diag/crs/rac122b/crs/trace/ocssd.trc.

此時,我們從集群的alert日志中看到由于節(jié)點1被集群驅(qū)逐了。


通過從節(jié)點2的ocssd.trc日志我們看到如下信息:

 

[grid@rac122b trace]$ cat ocssd.trc |egrep -i weight| tail -300

……省略部分信息

2018-01-25 09:44:18.347 :    CSSD:1494243072: clssnmrCheckNodeWeight: node(1) has weight stamp(0) pebbles (0) goldstars (0) flags (0) SpoolVersion (0)

2018-01-25 09:44:18.347 :    CSSD:1494243072: clssnmrCheckNodeWeight: node(2) has weight stamp(412336032) pebbles (0) goldstars (1) flags (3) SpoolVersion (0)

2018-01-25 09:44:18.347 :    CSSD:1494243072: clssnmrCheckNodeWeight: Server pool version not consistent

2018-01-25 09:44:18.347 :    CSSD:1494243072: clssnmrCheckNodeWeight: stamp(412336032), completed(1/2)

2018-01-25 09:44:18.347 :    CSSD:1494243072: clssnmrCheckSplit: Waiting for node weights, stamp(412336032)

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmrCheckNodeWeight: node(2) has weight stamp(412336032) pebbles (0) goldstars (1) flags (3) SpoolVersion (0)

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmrCheckNodeWeight: Server pool version not consistent

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmrCheckNodeWeight: stamp(412336032), completed(1/1)

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmCompareNodeWeights: Best map is same as the cohort map of the current node

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmFindBestMap: Using base map(2) of node(1) count(0), low(65535), bestcount(0), best_low(65535), cur_weightpebbles (0) goldstars (0) flags (0) SpoolVersion (0)best_weightpebbles (0) goldstars (0) flags (0) SpoolVersion (0)

2018-01-25 09:44:19.777 :    CSSD:1494243072: clssnmCompareNodeWeights: count(1), low(2), bestcount(0), best_low(65535), cur_weight: pebbles(0) goldstars(1) pubnw(1) flexasm(1)best_weight: pebbles(0) goldstars(0)pubnw(0) flexasm(0)


從上述信息來看,CSSD進程會去檢查節(jié)點的權(quán)重情況(CheckNodeWeight)以及比較節(jié)點的權(quán)重(CompareNodeWeights)節(jié)點2的權(quán)重大,此時,節(jié)點2這個子群戰(zhàn)勝了節(jié)點1子群,因而我們看到的情況是節(jié)點1被集群驅(qū)逐。


 

設置節(jié)點2上的service的權(quán)重


 

將心跳網(wǎng)絡恢復以及集群狀態(tài)恢復正常后,我們將對service級別的權(quán)重進行設置。


恢復集群server的權(quán)重:

 

[root@rac122b ~]# crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is yes.

[root@rac122b ~]# crsctl set server css_critical no

CRS-4416: Server attribute 'CSS_CRITICAL' successfully changed. Restart Oracle High Availability Services for new value to take effect.

[root@rac122b ~]#  crsctl get server css_critical

CRS-5092: Current value of the server attribute CSS_CRITICAL is no.

[root@rac122b ~]#


設置完成后,提示需要重啟OHAS才能生效。

 

$ srvctl stop instance -d milodb -i milodb2

# crsctl stop crs

# crsctl start crs


這里我們添加兩個專門用于做權(quán)重控制的服務(不對外使用):


srvctl add service -db milodb -service milodb_wt_srv1 -preferred milodb1

srvctl add service -db milodb -service milodb_wt_srv2 -preferred milodb2

srvctl start service -database milodb -service milodb_wt_srv1

srvctl start service -database milodb -service milodb_wt_srv2



在實例1上的服務不設置權(quán)重:

 

[oracle@rac122b ~]$ crsctl stat res ora.milodb.milodb_wt_srv1.svc -f | grep CSS

CSS_CRITICAL=no


在實例2上的服務設置權(quán)重:

 

[oracle@rac122b ~]$ crsctl stat res ora.milodb.milodb_wt_srv2.svc -f | grep CSS

CSS_CRITICAL=no

[oracle@rac122b ~]$ srvctl modify service -db milodb -service milodb_wt_srv2 -css_critical yes

[oracle@rac122b ~]$ crsctl stat res ora.milodb.milodb_wt_srv2.svc -f | grep CSS

CSS_CRITICAL=yes

[oracle@rac122b ~]$


此時,我們來觀察集群的一些日志情況:

 

節(jié)點1集群的alert日志:


2018-01-25 11:33:30.343 [OCSSD(3652)]CRS-1612: Network communication with node rac122b (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.910 seconds

2018-01-25 11:33:38.351 [OCSSD(3652)]CRS-1611: Network communication with node rac122b (2) missing for 75% of timeout interval.  Removal of this node from cluster in 6.900 seconds

2018-01-25 11:33:42.576 [OCSSD(3652)]CRS-1610: Network communication with node rac122b (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.680 seconds

2018-01-25 11:33:45.255 [OCSSD(3652)]CRS-1609: This node is unable to communicate with other nodes in the cluster and is going down to preserve cluster integrity; details at (:CSSNM00008:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc.

2018-01-25 11:33:45.255 [OCSSD(3652)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc

2018-01-25 11:33:45.269 [OCSSD(3652)]CRS-1652: Starting clean up of CRSD resources.

2018-01-25 11:33:47.289 [OCSSD(3652)]CRS-1608: This node was evicted by node 2, rac122b; details at (:CSSNM00005:) in /u01/app/grid/diag/crs/rac122a/crs/trace/ocssd.trc.


節(jié)點2集群的alert日志:


2018-01-25 11:33:29.844 [OCSSD(3983)]CRS-1612: Network communication with node rac122a (1) missing for 50% of timeout interval.  Removal of this node from cluster in 14.900 seconds

2018-01-25 11:33:37.946 [OCSSD(3983)]CRS-1611: Network communication with node rac122a (1) missing for 75% of timeout interval.  Removal of this node from cluster in 6.790 seconds

2018-01-25 11:33:41.947 [OCSSD(3983)]CRS-1610: Network communication with node rac122a (1) missing for 90% of timeout interval.  Removal of this node from cluster in 2.790 seconds

2018-01-25 11:33:46.684 [OCSSD(3983)]CRS-1607: Node rac122a is being evicted in cluster incarnation 412345406; details at (:CSSNM00007:) in /u01/app/grid/diag/crs/rac122b/crs/trace/ocssd.trc.

2018-01-25 11:33:47.755 [ORAAGENT(5634)]CRS-5818: Aborted command 'check' for resource 'ora.SYSTEMDG.dg'. Details at (:CRSAGF00113:) {0:1:11} in /u01/app/grid/diag/crs/rac122b/crs/trace/crsd_oraagent_grid.trc.

2018-01-25 11:33:47.759 [ORAAGENT(5634)]CRS-5818: Aborted command 'check' for resource 'ora.DATADG.dg'. Details at (:CRSAGF00113:) {0:1:11} in /u01/app/grid/diag/crs/rac122b/crs/trace/crsd_oraagent_grid.trc.

2018-01-25 11:33:49.250 [OCSSD(3983)]CRS-1601: CSSD Reconfiguration complete. Active nodes are rac122b .

2018-01-25 11:33:49.617 [CRSD(5349)]CRS-5504: Node down event reported for node 'rac122a'.


從節(jié)點2的ocssd日志看到,同樣由于節(jié)點2的權(quán)重高,因而最終將節(jié)點1驅(qū)逐除了集群:


 

 

后續(xù)測試將兩個服務都設置權(quán)重時,情況與不設置權(quán)重的情況一致,即節(jié)點1驅(qū)逐節(jié)點2。

總結(jié)

通過上述場景的測試,我們得知:


  • 基于權(quán)重的節(jié)點驅(qū)逐可以在集群出現(xiàn)腦裂時,通過控制減少消除一些影響;

  • 配置server方式的權(quán)重,需要修改配置后,重啟crs才能生效;

  • 通過服務的方式配置節(jié)點的權(quán)重不需要重啟資源,可以實現(xiàn)動態(tài)控制,更靈活。


當然,由于在該特性是12.2引入的,屬于新特性,可能會存在一些bug,建議在使用前應該經(jīng)過充分測試后使用。


作者介紹

羅雪原,云和恩墨南區(qū)交付技術顧問。

個人有著8年的Oracle技術支持經(jīng)驗,曾服務過金融、保險、電力、政府、運營商等客戶,有著較豐富的Troubleshooting以及優(yōu)化經(jīng)驗。

  • 1.公司登記注冊于2003年1月27日,清遠市桑達電子網(wǎng)絡媒體有限公司
    2.公司2006年起成為清遠市政府定點協(xié)議供貨商,電子采購供貨商
    3.公司2007年被清遠市相關政府部門評為安防行業(yè)狀元
    4.公司2007年起成為長城電腦清遠如意服務站(SP368)
    5.公司2007年承建清遠市橫河路口電子警察工程,開創(chuàng)清遠電子警察先河。
  • 6.公司2007年起成為IBM合作伙伴、公司2010年底成為金蝶軟件清遠金牌代理(伙伴編號:30030013)
    7.公司組團隊參加南方都市報組織的創(chuàng)富評選,獲廣東80強。公司申請多項軟件著作權(quán)、專利權(quán)
    8.2016年起公司成為粵東西北地區(qū)為數(shù)不多的雙軟企業(yè),確立“讓軟件驅(qū)動世界,讓智能改變生活!"企業(yè)理想
    9.2016-01-29更名為廣東互動電子網(wǎng)絡媒體有限公司
    10.2021-01-13更名為廣東互動電子有限公司
  • 投資合作咨詢熱線電話:0763-3391888 3323588
  • 做一個負責任的百年企業(yè)! 天行健,君子以自強不息;地勢坤,君子以厚德載物;
    為用戶創(chuàng)造價值! 讓軟件驅(qū)動世界; 讓智能改變生活; 超越顧客期望,幫助顧客成功;
    對客戶負責,對員工負責,對企業(yè)命運負責!幫助支持公司的客戶成功;幫助忠誠于公司的員工成功!
  • 聯(lián)系電話:0763-3391888 3323588 3318977
    服務熱線:18023314222 QQ:529623964
  • 工作QQ:2501204690 商務QQ: 602045550
    投資及業(yè)務投訴QQ: 529623964
    微信:小米哥 微信號:qysed3391888
    騰訊微博:桑達網(wǎng)絡-基石與起點
  • E-MAIL:222#QYSED.CN ok3391888#163.com (請用@替換#)
在線客服
  • 系統(tǒng)集成咨詢
    點擊這里給我發(fā)消息
  • 網(wǎng)站\微信\軟件咨詢
    點擊這里給我發(fā)消息
  • 售后服務
    點擊這里給我發(fā)消息
  • 投資合作
    點擊這里給我發(fā)消息