错误信息

  1. 2020-03-28 08:24:17.555175 CST,,,p22500,th-1680533376,,,,0,,,seg2,,,,,"FATAL","XX000","
  2. could not create semaphores: No space left on device (pg_sema.c:126)","Failed system call
  3. was semget(7000031, 17, 03600).","This error does *not* mean that you have run out of disk 
  4. space.  It occurs when either the system limit for the maximum number of semaphore sets 
  5. (SEMMNI), or the system wide maximum number of semaphores (SEMMNS), would be exceeded.  
  6. You need to raise the respective kernel parameter.  Alternatively, reduce PostgreSQL's 
  7. consumption of semaphores by reducing its max_connections parameter.
  8. The PostgreSQL documentation contains more information about configuring your system for PostgreSQL.",,,,,,"InternalIpcSemaphoreCreate","pg_sema.c",126,1    0x9f9b7c postgres errstart (elog.c:557)
  9. 2    0x807448 postgres PGSemaphoreCreate (pg_sema.c:113)
  10. 3    0x87d5da postgres InitProcGlobal (proc.c:259)
  11. 4    0x869765 postgres CreateSharedMemoryAndSemaphores (ipci.c:288)
  12. 5    0x81b97b postgres PostmasterMain (postmaster.c:1327)
  13. 6    0x4cf0e1 postgres main (main.c:205)
  14. 7    0x7fe8976c03d5 libc.so.6 __libc_start_main + 0xf5
  15. 8    0x4cf6bc postgres <symbol not found> + 0x4cf6bc

错误问题描述

初始化集群后，4台镜像问题没有启动

修改思路

排查失败镜像启动日志

查找结果

在opt/greenplum/data1/mirror/gpseg2/pg_log/startup.log
文件中有报错信息
其他镜像节点报错相同

修改结果

根据报错信息，判定是系统参数中kernel.sem 值不够导致的，解决方式是提高信号量。

  1. vi /etc/sysctl.conf
  2. 修改
  3. kernel.sem = 500 2048000 200 40960
  4. 改为
  5. kernel.sem = 32000 1024000000 500 32000
  6. 执行
  7. sysctl -p  每个服务器都修改
  9. SEMMSL: 每组需要多少信号量, 代表每个信号集中的最大信号量数目 。
  10. SEMMNS: 需要多少信号量,     代表系统范围内的最大信号量总数目(第二列 = 第一列 * 第四列 )。
  11. SEMOPM: 代表每个信号发生时的最大系统操作数目。
  12. SEMMNI: 需要多少组, 代表系统范围内的最大信号集总数目。

修改后使用恢复命令 恢复数据库
或者如果 第一次初始化数据库 也可以执行删除所有数据库重新初始化
一下是删除数据库信息操作

  1. 清除初始化系统信息： rm -rf /home/gpadmin/gpAdminLogs/*
  2. 杀掉起来的greenplum进程：pkill postgres
  3. rm -rf /opt/greenplum/data/master/gpseg-1
  4. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data1/primary/gpseg0"
  5. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data2/primary/gpseg1"
  6. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data1/primary/gpseg2"
  7. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data2/primary/gpseg3"
  8. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data1/mirror/gpseg0"
  9. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data2/mirror/gpseg1"
  10. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data1/mirror/gpseg2"
  11. gpssh -f /usr/local/greenplum-db/all_hosts -e "rm -rf  /opt/greenplum/data2/mirror/gpseg3"