Jump to content

HTTPS/testing

From Wikitech
< HTTPS
(Redirected from Https/testing)

Testing puppet changes to ssl terminators (read and sanity check before blindly following this, maybe something's changed since this was written)

Note: puppet agent --disable didn't actually disable puppet on two of the 12 hosts last time we did this, no idea why.

  1. verify that /usr/local/dsh/node_groups/ssl is in sync with /home/w/conf/pybal/*/https on fenari.
  2. take one ssl server out of pool:
    • we want to take one out from tampa since it will have minimal impact on traffic
    • go to /home/wikipedia/common/docroot/noc/pybal/pmtpa, edit the https file
    • change one of them from 'enabled' : True to 'enabled' : False
    • wait a minute for the changes to propogate
    • check that there is no traffic going to the host; ssh on and do netstat -tanvp (t is tcp, p gives pid and program name, dunno if we care about that but whatever) and check that there is no traffic from outside our subnets
  3. turn off puppet on all ssl hosts, this is esams *and* tampa *and* eqiad
    • on fenari dsh -g ssl -cM 'puppet agent --disable'
    • check one of them to be sure it really won't run (ssh over and try puppet agent --onetime --verbose --no-daemonize and make sure it whines)
  4. on the depooled host re-enable puppet for testing
    • puppet agent --enable
  5. on the depooled host run puppet to get the changes over, see how nginx is
    • puppetd --test
    • check that nginx is running, check the logs: /var/log/nginx/error.log and /var/log/nginx/access.log and expect nothing much interesting in them
    • check one of the nginx conf files for sanity: /etc/nginx/sites-enabled/wikipedia (it cn be compared to another host)
    • restart nginx on the host
  6. test some urls on the server
    • you have to run from the server itself
    • you have to have the service ip in the url (so it's intercepted by the listener on the lo port)
    • probably need to install curl on there (don't forget to remove after done)
    • curl -k -I -v -H "Host: en.wikipedia.org" https://208.80.152.201/wiki/
    • curl -k -I -v -H "Host: en.wikipedia.org" https://208.80.152.201/wiki/Main_Page
    • (address stolen from looking at VIP which it listens to in /etc/nginx/sites-enabled/wikipedia
  7. re-enable and run puppet on a host in a cluster with some traffic, reload nginx, see how things are
  8. re-enable and run puppet and reload nginx on a couple more boxes, say the rest in $cluster-with-some-traffic one at a time
  9. If it all looks right enable puppet everywhere
    • on fenari dsh -g ssl -cM 'puppet agent --enable'
    • make sure puppet has run everywhere...(could do via dsh)
    • restart nginx on all hosts dsh -g ssl -M -w '/etc/init.d/nginx reload; sleep 60'

This is what a successful test looks like: 

root@ssl4:~# curl -k -I -v -H 'Host: en.wikipedia.org' https://208.80.152.201/wiki/
* About to connect() to 208.80.152.201 port 443 (#0)
*   Trying 208.80.152.201... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*      subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
*      start date: 2012-10-22 00:00:00 GMT
*      expire date: 2016-01-20 12:00:00 GMT
*      common name: *.wikipedia.org (does not match '208.80.152.201')
*      issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
*      SSL certificate verify ok.
> HEAD /wiki/ HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: en.wikipedia.org
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: nginx/1.1.19
Server: nginx/1.1.19
< Date: Thu, 09 May 2013 06:06:10 GMT
Date: Thu, 09 May 2013 06:06:10 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Connection: keep-alive
Connection: keep-alive
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Cache-Control: s-maxage=1200, must-revalidate, max-age=0
Cache-Control: s-maxage=1200, must-revalidate, max-age=0
< Vary: Accept-Encoding,X-Forwarded-Proto,Cookie
Vary: Accept-Encoding,X-Forwarded-Proto,Cookie
< X-Vary-Options: Accept-Encoding;list-contains=gzip,X-Forwarded-Proto,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
X-Vary-Options: Accept-Encoding;list-contains=gzip,X-Forwarded-Proto,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
< Last-Modified: Thu, 09 May 2013 06:06:10 GMT
Last-Modified: Thu, 09 May 2013 06:06:10 GMT
< Location: https://en.wikipedia.org/wiki/Main_Page
Location: https://en.wikipedia.org/wiki/Main_Page
< X-Cache: MISS from sq76.wikimedia.org
X-Cache: MISS from sq76.wikimedia.org
< X-Cache-Lookup: HIT from sq76.wikimedia.org:3128
X-Cache-Lookup: HIT from sq76.wikimedia.org:3128
< X-Cache: MISS from sq66.wikimedia.org
X-Cache: MISS from sq66.wikimedia.org
< X-Cache-Lookup: MISS from sq66.wikimedia.org:80
X-Cache-Lookup: MISS from sq66.wikimedia.org:80
< Via: 1.1 sq76.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq66.wikimedia.org:80 (squid/2.7.STABLE9)
Via: 1.1 sq76.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq66.wikimedia.org:80 (squid/2.7.STABLE9)
* no chunk, no close, no size. Assume close to signal end
< 
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
root@ssl4:~# curl -k -I -v -H 'Host: en.wikipedia.org' https://208.80.152.201/wiki/Main_Page
* About to connect() to 208.80.152.201 port 443 (#0)
*   Trying 208.80.152.201... connected
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
*      subject: C=US; ST=California; L=San Francisco; O=Wikimedia Foundation, Inc.; CN=*.wikipedia.org
*      start date: 2012-10-22 00:00:00 GMT
*      expire date: 2016-01-20 12:00:00 GMT
*      common name: *.wikipedia.org (does not match '208.80.152.201')
*      issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert High Assurance CA-3
*      SSL certificate verify ok.
> HEAD /wiki/Main_Page HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Accept: */*
> Host: en.wikipedia.org
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.1.19
Server: nginx/1.1.19
< Date: Thu, 09 May 2013 06:09:53 GMT
Date: Thu, 09 May 2013 06:09:53 GMT
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Content-Length: 64909
Content-Length: 64909
< Connection: keep-alive
Connection: keep-alive
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< Cache-Control: s-maxage=2678400, must-revalidate, max-age=0
Cache-Control: s-maxage=2678400, must-revalidate, max-age=0
< Content-Language: en
Content-Language: en
< Vary: Accept-Encoding,Cookie
Vary: Accept-Encoding,Cookie
< X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
X-Vary-Options: Accept-Encoding;list-contains=gzip,Cookie;string-contains=enwikiToken;string-contains=enwikiLoggedOut;string-contains=enwiki_session;string-contains=centralauth_Token;string-contains=centralauth_Session;string-contains=centralauth_LoggedOut;string-contains=mf_useformat
< Last-Modified: Thu, 09 May 2013 05:50:28 GMT
Last-Modified: Thu, 09 May 2013 05:50:28 GMT
< X-Cache: MISS from sq75.wikimedia.org
X-Cache: MISS from sq75.wikimedia.org
< X-Cache-Lookup: MISS from sq75.wikimedia.org:3128
X-Cache-Lookup: MISS from sq75.wikimedia.org:3128
< Age: 1164
Age: 1164
< X-Cache: HIT from sq75.wikimedia.org
X-Cache: HIT from sq75.wikimedia.org
< X-Cache-Lookup: HIT from sq75.wikimedia.org:80
X-Cache-Lookup: HIT from sq75.wikimedia.org:80
< Via: 1.1 sq75.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq75.wikimedia.org:80 (squid/2.7.STABLE9)
Via: 1.1 sq75.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0 sq75.wikimedia.org:80 (squid/2.7.STABLE9)
< 
* Connection #0 to host 208.80.152.201 left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=HTTPS/testing&oldid=81385"