数据库启动不了,一直报 the dat...
 
通知
清除全部

数据库启动不了,一直报 the database system is starting up


gpfen
帖子: 3
Topic starter
初出茅庐
已加入: 1年 前

经过是这样的,集群正常运行的情况下,发现master  standby未启动,于是执行了 gpinitstandby  -n  将standby启动。 紧接着又执行了  gpstop  -M fast。    在确定所有gp进程都停止了的情况下,执行gpstart  -a或gpstart  -m启动数据库的时候一直 报 the database system is starting up。 启动不了

 

其中执行gpstop  -M fast的日志如下:

20201208069026 gpstop-Starting gpstop with args: -M fast
20201208069026 gpstop-Gathering information and validating the environment...
20201208069026 gpstop-Obtaining Greenplum Master catalog information
20201208069026 gpstop-Obtaining Segment details from master...
20201208069026 gpstop-Greenplum Version: 'postgres (Greenplum Database) 6.9.0 build commit:ef010af28862a0fed172ca96620cc1037aac71a0'
20201208069026 gpstop---------------------------------------------
20201208069026 gpstop-Master instance parameters
20201208069026 gpstop---------------------------------------------
20201208069026 gpstop- Master Greenplum instance process active PID = 72190
20201208069026 gpstop- Database = template1
20201208069026 gpstop- Master port = 5432
20201208069026 gpstop- Master directory = /usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208069026 gpstop- Shutdown mode = fast
20201208069026 gpstop- Timeout = 120
20201208069026 gpstop- Shutdown Master standby host = On
20201208069026 gpstop---------------------------------------------
20201208069026 gpstop-Segment instances that will be shutdown:
20201208069026 gpstop---------------------------------------------
20201208069026 gpstop- Host Datadir Port Status
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatap1/gpseg0 6000 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatam1/gpseg0 7000 u
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatap2/gpseg1 6001 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatam2/gpseg1 7001 u
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatap3/gpseg2 6002 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatam3/gpseg2 7002 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatap1/gpseg3 6000 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatam1/gpseg3 7000 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatap2/gpseg4 6001 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatam2/gpseg4 7001 u
20201208069026 gpstop- CN2 /usr/zmp_pc/gp6data/gpdatap3/gpseg5 6002 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatam3/gpseg5 7002 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatap1/gpseg6 6000 u
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatam1/gpseg6 7000 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatap2/gpseg7 6001 u
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatam2/gpseg7 7001 u
20201208069026 gpstop- CN3 /usr/zmp_pc/gp6data/gpdatap3/gpseg8 6002 u
20201208069026 gpstop- CN1 /usr/zmp_pc/gp6data/gpdatam3/gpseg8 7002 u
20201208069026 gpstop-Commencing Master instance shutdown with mode='fast'
20201208069026 gpstop-Master segment instance directory=/usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208069026 gpstop-Attempting forceful termination of any leftover master process
20201208069026 gpstop-Terminating processes for segment /usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208069026 gpstop-Failed to kill processes for segment /usr/zmp_pc/gp6data/gpmaster/gpseg-1: ([Errno 3] No such process)
20201208069026 gpstop-Stopping master standby host CN2 mode=fast
20201208069026 gpstop-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occurred. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 CN2 ". /usr/local/greenplum-db-6.9.0/greenplum_path.sh; $GPHOME/bin/pg_ctl -D /usr/zmp_pc/gp6data/gpmaster/gpseg-1 -m fast -w -t 120 stop"' cmd had rc=1 completed=True halted=False
stdout=''
stderr='pg_ctl: PID file "/usr/zmp_pc/gp6data/gpmaster/gpseg-1/postmaster.pid" does not exist
Is server running?
'
20201208069026 gpstop-Successfully shutdown standby process on CN2
20201208069026 gpstop-Targeting dbid [2, 11, 3, 12, 4, 13, 5, 14, 6, 15, 7, 16, 8, 17, 9, 18, 10, 19] for shutdown
20201208069026 gpstop-Commencing parallel primary segment instance shutdown, please wait...
20201208069026 gpstop-0.00% of jobs completed
20201208069026 gpstop-66.67% of jobs completed
20201208069026 gpstop-100.00% of jobs completed
20201208069026 gpstop-Commencing parallel mirror segment instance shutdown, please wait...
20201208069026 gpstop-0.00% of jobs completed
20201208069026 gpstop-100.00% of jobs completed
20201208069026 gpstop-----------------------------------------------------
20201208069026 gpstop- Segments stopped successfully = 18
20201208069026 gpstop- Segments with errors during stop = 0
20201208069026 gpstop-----------------------------------------------------
20201208069026 gpstop-Successfully shutdown 18 of 18 segment instances
20201208069026 gpstop-Database successfully shutdown with no errors reported
20201208069026 gpstop-Cleaning up leftover gpmmon process
20201208069026 gpstop-No leftover gpmmon process found
20201208069026 gpstop-Cleaning up leftover gpsmon processes
20201208069026 gpstop-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20201208069026 gpstop-Cleaning up leftover shared memory
20201208211954 gpstop-Starting gpstop with args: -M fast
20201208211954 gpstop-Gathering information and validating the environment...
20201208211954 gpstop-Obtaining Greenplum Master catalog information
20201208211954 gpstop-Obtaining Segment details from master...
20201208211954 gpstop-gpstop failed. (Reason='FATAL: the database system is starting up
DETAIL: last replayed record at 0/0
') exiting... 

 

执行gpstart 的日志如下:

20201208002303 gpstart-Starting gpstart with args: -m -v
20201208002303 gpstart-Setting level of parallelism to: 64
20201208002303 gpstart-Gathering information and validating the environment...
20201208002303 gpstart---Checking that current user can use GP binaries
20201208002303 gpstart-Obtaining master's port from master data directory
20201208002303 gpstart-Read from postgresql.conf port=5432
20201208002303 gpstart-Read from postgresql.conf max_connections=250
20201208002303 gpstart-Running Command: $GPHOME/bin/postgres --gp-version
20201208002303 gpstart-Greenplum Binary Version: 'postgres (Greenplum Database) 6.9.0 build commit:ef010af28862a0fed172ca96620cc1037aac71a0'
20201208002303 gpstart-Running Command: $GPHOME/bin/postgres --catalog-version
20201208002303 gpstart-Greenplum Catalog Version: '301908232'
20201208002303 gpstart-Running Command: $GPHOME/bin/pg_controldata /usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208002303 gpstart-Check if Master is already running...
20201208002303 gpstart-*************************************************************************
20201208002303 gpstart-Master-only start requested. Disruptive action if standby master configured.
20201208002303 gpstart-This is advisable only under the direct supervision of Greenplum support.
20201208002303 gpstart-This mode of operation is not supported in a production environment and
20201208002303 gpstart-may lead to a split-brain condition and possible unrecoverable data loss.
20201208002303 gpstart-
**********************************************************************
20201208002303 gpstart-Starting Master instance in admin mode
20201208002303 gpstart-Running Command: env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /usr/zmp_pc/gp6data/gpmaster/gpseg-1 -l /usr/zmp_pc/gp6data/gpmaster/gpseg-1/pg_log/startup.log -w -t 600 -o " -p 5432 -c gp_role=utility " start
20201208002303 gpstart-Obtaining Greenplum Master catalog information
20201208002303 gpstart-Obtaining Segment details from master...
20201208002303 gpstart-Connecting to dbname='template1'
20201208002303 gpstart-gpstart failed. exiting...
Traceback (most recent call last):
File "/usr/local/greenplum-db-6.9.0/lib/python/gppylib/mainUtils.py", line 270, in simple_main_locked
exitCode = commandObject.run()
File "/usr/local/greenplum-db-6.9.0/bin/gpstart", line 110, in run
self._startMaster()
File "/usr/local/greenplum-db-6.9.0/bin/gpstart", line 393, in _startMaster
self.gparray = GpArray.initFromCatalog(self.dburl, utility=True)
File "/usr/local/greenplum-db-6.9.0/lib/python/gppylib/gparray.py", line 958, in initFromCatalog
conn = dbconn.connect(dbURL, utility)
File "/usr/local/greenplum-db-6.9.0/lib/python/gppylib/db/dbconn.py", line 206, in connect
cnx = pgdb._connect_(cstr, dbhost, dbport, dbopt, dbtty, dbuser, dbpasswd)
InternalError: FATAL: the database system is starting up
DETAIL: last replayed record at 0/0

20201208005420 gpstart-Starting gpstart with args: -m
20201208005420 gpstart-Gathering information and validating the environment...
20201208005420 gpstart-Greenplum Binary Version: 'postgres (Greenplum Database) 6.9.0 build commit:ef010af28862a0fed172ca96620cc1037aac71a0'
20201208005420 gpstart-Greenplum Catalog Version: '301908232'
20201208005420 gpstart-**********************************************************************
20201208005420 gpstart-Master-only start requested. Disruptive action if standby master configured.
20201208005420 gpstart-This is advisable only under the direct supervision of Greenplum support.
20201208005420 gpstart-This mode of operation is not supported in a production environment and
20201208005420 gpstart-may lead to a split-brain condition and possible unrecoverable data loss.
20201208005420 gpstart-
*************************************************************************
20201208005420 gpstart-Starting Master instance in admin mode
20201208005420 gpstart-Failed to start Master instance in admin mode
20201208005420 gpstart-Error occurred: non-zero rc: -2
Command was: 'env GPSESSID=0000000000 GPERA=None $GPHOME/bin/pg_ctl -D /usr/zmp_pc/gp6data/gpmaster/gpseg-1 -l /usr/zmp_pc/gp6data/gpmaster/gpseg-1/pg_log/startup.log -w -t 600 -o " -p 5432 -c gp_role=utility " start'
rc=-2, stdout='waiting for server to start...................................................

1 Reply
xiaoxu123
帖子: 4
初出茅庐
已加入: 1年 前

从日志中可以看出,master先关闭了

发布者: @gpfen

20201208069026 gpstop-Master segment instance directory=/usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208069026 gpstop-Attempting forceful termination of any leftover master process
20201208069026 gpstop-Terminating processes for segment /usr/zmp_pc/gp6data/gpmaster/gpseg-1
20201208069026 gpstop-Failed to kill processes for segment /usr/zmp_pc/gp6data/gpmaster/gpseg-1: ([Errno 3] No such process)

 

回复
Close Bitnami banner
Bitnami