坑爹的Failover (pgpool)

pgpool是一个很不错的中间件，具体安装配置可参考PostGreSQL pgpool-II集群配置，不过如果你没有很深入的去了解，那么会有踩不完的坑。今天我们聊聊遇到的pgpool中关于failover的坑。failover这个功能初衷是好的.当主库发生问题时，自动切换到备库，简直不要太爽。不过某些时候我们不想让它切过去，这就有点难为情了。

kill 头锁进程

某些情况，数据库发生大面积锁，导致业务不能继续，那么你要找到导致这么一堆锁发生的原因，也就是头锁，然后干掉它。这里有一个sql 可以查看当前数据库锁的情况，并根据锁的先后顺序做了排序，排在首位的就是头锁。为了简单，我们可以创建成视图v_locks_monitor，然后直接用select * from v_locks_monitor 进行查看。
创建视图

create view v_locks_monitor as   
with    
t_wait as    
(    
  select a.mode,a.locktype,a.database,a.relation,a.page,a.tuple,a.classid,a.granted,   
  a.objid,a.objsubid,a.pid,a.virtualtransaction,a.virtualxid,a.transactionid,a.fastpath,    
  b.state,b.query,b.xact_start,b.query_start,b.usename,b.datname,b.client_addr,b.client_port,b.application_name   
    from pg_locks a,pg_stat_activity b where a.pid=b.pid and not a.granted   
),   
t_run as   
(   
  select a.mode,a.locktype,a.database,a.relation,a.page,a.tuple,a.classid,a.granted,   
  a.objid,a.objsubid,a.pid,a.virtualtransaction,a.virtualxid,a.transactionid,a.fastpath,   
  b.state,b.query,b.xact_start,b.query_start,b.usename,b.datname,b.client_addr,b.client_port,b.application_name   
    from pg_locks a,pg_stat_activity b where a.pid=b.pid and a.granted   
),   
t_overlap as   
(   
  select r.* from t_wait w join t_run r on   
  (   
    r.locktype is not distinct from w.locktype and   
    r.database is not distinct from w.database and   
    r.relation is not distinct from w.relation and   
    r.page is not distinct from w.page and   
    r.tuple is not distinct from w.tuple and   
    r.virtualxid is not distinct from w.virtualxid and   
    r.transactionid is not distinct from w.transactionid and   
    r.classid is not distinct from w.classid and   
    r.objid is not distinct from w.objid and   
    r.objsubid is not distinct from w.objsubid and   
    r.pid <> w.pid   
  )    
),    
t_unionall as    
(    
  select r.* from t_overlap r    
  union all    
  select w.* from t_wait w    
)    
select locktype,datname,relation::regclass,page,tuple,virtualxid,transactionid::text,classid::regclass,objid,objsubid,   
string_agg(   
'Pid: '||case when pid is null then 'NULL' else pid::text end||chr(10)||   
'Lock_Granted: '||case when granted is null then 'NULL' else granted::text end||' , Mode: '||case when mode is null then 'NULL' else mode::text end||' , FastPath: '||case when fastpath is null then 'NULL' else fastpath::text end||' , VirtualTransaction: '||case when virtualtransaction is null then 'NULL' else virtualtransaction::text end||' , Session_State: '||case when state is null then 'NULL' else state::text end||chr(10)||   
'Username: '||case when usename is null then 'NULL' else usename::text end||' , Database: '||case when datname is null then 'NULL' else datname::text end||' , Client_Addr: '||case when client_addr is null then 'NULL' else client_addr::text end||' , Client_Port: '||case when client_port is null then 'NULL' else client_port::text end||' , Application_Name: '||case when application_name is null then 'NULL' else application_name::text end||chr(10)||    
'Xact_Start: '||case when xact_start is null then 'NULL' else xact_start::text end||' , Query_Start: '||case when query_start is null then 'NULL' else query_start::text end||' , Xact_Elapse: '||case when (now()-xact_start) is null then 'NULL' else (now()-xact_start)::text end||' , Query_Elapse: '||case when (now()-query_start) is null then 'NULL' else (now()-query_start)::text end||chr(10)||    
'SQL (Current SQL in Transaction): '||chr(10)||  
case when query is null then 'NULL' else query::text end,    
chr(10)||'--------'||chr(10)    
order by    
  (  case mode    
    when 'INVALID' then 0   
    when 'AccessShareLock' then 1   
    when 'RowShareLock' then 2   
    when 'RowExclusiveLock' then 3   
    when 'ShareUpdateExclusiveLock' then 4   
    when 'ShareLock' then 5   
    when 'ShareRowExclusiveLock' then 6   
    when 'ExclusiveLock' then 7   
    when 'AccessExclusiveLock' then 8   
    else 0   
  end  ) desc,   
  (case when granted then 0 else 1 end)  
) as lock_conflict  
from t_unionall   
group by   
locktype,datname,relation,page,tuple,virtualxid,transactionid::text,classid,objid,objsubid ;

此处不举例，我们重点是pgpool。

坑爹的参数failover_on_backend_eror

重点来了，我们kill了一个头锁，突然发现，链接被重置了，再看，主库变备库，备库变主库了。。。
我们来看下pgpool官网关于Failover and Failback 这段的描述。开头定义了什么是Failover: pgpool发现后端连接的pg节点不可用后自动剔除掉该节点。然后调用自动以的脚本去做一些事情，一般是自动切换
注意我们红框标注的这个地方，提到了坑爹参数 failover_on_backend_eror 。该参数如果是off 状态，那么当连接到pg的连接发生错误或者巴拉巴拉的什么情况，pgpool 仅仅是断开这个session的连接。那么就不会发生我们kill掉一个进程从而导致failover的情况

问题来了，看这个参数的解释，这里没有提默认是on还是off。但是实际配置上，我们一般不去了解所有的参数的意义，都是遵循大部分的默认配置。当我们kill掉一个连接，发生了自动切换后，我们知道了，参数默认是on。

最佳实践

如果你不想因为kill 一个头锁的进程而导致pgpool自动切，那么一定要注意显性的配置该参数为off。
关于防止一般问题导致自动切换，可以多注意下health check的几个参数，降低pgpool对pg的敏感度。根据名字也知道是什么意思，详细可参考 https://www.pgpool.net/docs/latest/en/html/runtime-config-health-check.html.

health_check_max_retries = 3
health_check_retry_delay = 3