Puppetmaster高可用和可扩展的方案设计

时间:2022-07-08 15:47:30

Puppet是当前devops中常用于管理系统配置和应用部署,多数会使用其C/S架构的方式来进行部署,其中puppetmaster是集群中配置管理的核心节点。在实际的生产环境中,如果因为master节点性能不够或者发生意外宕机,可能会影响到实际业务,因此维护一个高可用和可扩展的puppetmaster池子是一个首要任务。

这里我使用了一种常规的方案:前端使用apache/nginx做负载均衡,使用packmaker/keepalived来做健康检查和故障切换,来做HA,后端起多个puppetmaster实例做横向扩展,来提高处理能力。

方案验证

这里,我将在在每台Master Node上起两个puppetmaster实例,前端使用Apache作负载均衡,keepalived做健康检查。唯一的难点是证书同步问题,在部署中将把我们将证书设成自动认证,只接受fqdn是*.clustername.ustack.com的机器,就不需同步证书了。

IP 主机名 角色 vip
192.168.1.53 ha-puppet1.ustack.com puppet master 192.168.1.103
192.168.1.54 ha-puppet2.ustack.com puppet master 192.168.1.104

配置细节

客户端 配置文件

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
pluginsync=true

[agent]
server = ha-puppet.ustack.com
report = true
pluginsync = true
listen = true
runinterval = 300

Master node的配置选项

[master]
autosign = $confdir/autosign.conf { mode = 664 }

autosign.conf

*.ustack.com

服务器端配置信息

192.168.2.53

loadbalancer配置

apache的proxy监听在8140端口,后面可以配置多个puppetmaster进程

<Proxy balancer://puppetmaster>
BalancerMember http://127.0.0.1:18140
BalancerMember http://127.0.0.1:18141
</Proxy>

Listen 8140
<VirtualHost *:8140>

SSLEngine on
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

SSLCertificateFile /var/lib/puppet/ssl/certs/ha-puppet.ustack.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/ha-puppet.ustack.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
# CRL checking should be enabled; if you have problems with Apache complaining about the CRL, disable the next line
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars +ExportCertData

# The following client headers allow the same configuration to work with Pound.
RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e
RequestHeader unset X-Forwarded-For

# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120

RackAutoDetect On
DocumentRoot /etc/puppet/rack/public/
<Directory /etc/puppet/rack>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
<Location />
SetHandler balancer-manager
Order allow,deny
Allow from all
</Location>

ProxyPass / balancer://puppetmaster/
ProxyPassReverse / balancer://puppetmaster/
ProxyPreserveHost On

CustomLog /var/log/httpd/balance-8140-access.log combined
ErrorLog /var/log/httpd/balance-8140-error.log
CustomLog /var/log/httpd/balancer_ssl_requests.log "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>

第一个puppetmaster的vhost配置文件,puppetmaster实例的数量可以水平扩展:

Listen 18140
<VirtualHost *:18140>

SSLEngine off

# The following client headers allow the same configuration to work with Pound.
SetEnvIf set X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
SetEnvIf set X-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

# you probably want to tune these settings
PassengerHighPerformance on
PassengerMaxPoolSize 12
PassengerPoolIdleTime 1500
# PassengerMaxRequests 1000
PassengerStatThrottleRate 120

RackAutoDetect On
DocumentRoot /etc/puppet/rack/18140/public/
<Directory /etc/puppet/rack/18140/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
CustomLog /var/log/httpd/puppetmaster-18140-access.log combined
ErrorLog /var/log/httpd/puppetmaster-18140-error.log
</VirtualHost>

rack配置文件

拷贝rack配置文件给第一个puppetmaster:

rsync -avxH /etc/puppet/rack/{,18140}/

Keepalived配置文件

! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc

}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server localhost
smtp_connect_timeout 30
router_id route-45
}

vrrp_script chk_http_port {
script "/usr/bin/killall -0 httpd"
interval 2
weight 2
}

vrrp_instance 45 {
virtual_router_id 45


priority 100
state BACKUP

interface eth0

virtual_ipaddress {
192.168.2.103
}

track_script {
chk_http_port
}
}

vrrp_instance 46 {
virtual_router_id 46


priority 101
state MASTER

interface eth0

virtual_ipaddress {
192.168.2.104
}

track_script {
chk_http_port
}
}

192.168.2.54

apache的配置文件和53相同,唯一区别就是keepalived的配置文件上的IP地址互为热备,在此就不再赘述。

验证

在dnspod上绑定192.168.1.103/4到ha-puppet.ustack.com。

并在ha-puppet1上使用以下manifests文件:

node /default/ {

notify {'Hello,I am Master 1':}
}

node 'nginx.novalocal' inherits default{}

在ha-puppet2上使用以下manifests文件:

node /default/ {

notify {'Hello,I am Master 2':}
}

node 'nginx.novalocal' inherits default{}

测试结果,前两次是:

notice: Hello,I am Master 2
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 2]/message: defined 'message' as 'Hello,I am Master 2'
notice: Finished catalog run in 0.12 seconds

随后出现:

notice: Hello,I am Master 1
notice: /Stage[main]//Node[default]/Notify[Hello,I am Master 1]/message: defined 'message' as 'Hello,I am Master 1'
notice: Finished catalog run in 0.10 seconds

结论

以上方案验证通过,手动部署很简单,难点在于把它们设计成puppet module来进行部署时,需要考虑到所有的服务器需要使用同一个证书,因此需要在启动第一台puppetmaster的时候,根据设定的fqdn生成证书,然后修改hostname,同时把证书和配置文件同步到其他服务器上去。