Data Platform/Systems/Kerberos
Hadoop by default does not ship with a strong authentication mechanism for users and daemons. In order to enable its "Secure mode", an external authentication service must be plugged in, and the only compatible one is Kerberos.
When enabled, it means that users and daemons will need to authenticate to our Kerberos service before being able to use Hadoop. Please read the next sections to get more info about what do to.
High level overview
This diagram is a high level overview of how Kerberos authentication affects users. First of all, notice that the Hadoop cluster is the only part of the infrastructure that will be configured to use Kerberos. The red lines show which systems are required to authenticate with Kerberos in order to use Hadoop:
- Druid, since its deep storage is Hadoop HDFS. Please note that this will not mean that Druid itself will require Kerberos authentication from users, but only that Druid itself will need to authenticate before fetching data from HDFS. This means that Superset and Turnilo dashboards will keep working as before, without changes.
- Users on the Analytics Clients. Anyone who uses a tool on the clients that interacts with Hadoop (such as Oozie, Hive, Spark, Jupyter Notebooks) will need to authenticate via Kerberos.
How do I..
Authenticate via Kerberos
Run the kinit command, enter your password and then execute any command (spark, etc..). This is very important since if you don't do it, you'll see horrible error messages reported by basically anything you'll use. The kinit command grants you a so called Kerberos TGT (Ticket Granting Ticket), that will be used to allow you to authenticate to various services and hosts.
The ticket lasts for 2 days, so you will not need to run kinit every time, just every time it expires. You can inspect the status of your ticket via klist.
If you are using the analytics client nodes, then the Kerberos Auto-Renewal Service means that you are only required to enter your password once every 14 days.
Renewing a Ticket
Your Kerberos ticket expires after 2 days, but within this period it can be renewed and its lifetime extended by another 2 days using the command kinit -R
The maximum lifetime of a Kerberos ticket is currently configured to be 14 days, after which it cannot be renewed and you will have to request a new ticket with kinit and enter your Kerberos password.
Kerberos Auto-Renewal Service
The analytics client nodes have a system in place that will automatically renew your Kerberos ticket every day, up to the maximum permissible lifetime of the ticket.
Once you have run kinit for the first time and have a valid ticket, the next time you SSH into the same host a periodic task will be created for you that will renew your ticket every day at midnight UTC.
You will see a message like this:
You have a valid Kerberos ticket. Creating automatic Kerberos ticket renewal service.
On subsequent logins you will see a message like this:
You have a valid Kerberos ticket. Your automatic Kerberos ticket renewal service is also active on this host.
The result is that on these hosts you only have to enter your Kerberos password every 14 days, as opposed to every 2 days.
Get a password for Kerberos
Please create a Phabricator task with the "Data Engineering" tag to request a Kerberos identity. Check that:
- Your shell username is in analytics-privatedata-users.
- Your shell username and email are listed in the task's description.
When your request is granted, you'll receive an email containing a temporary password, that you'll be required to change during your first authentication.
If you have any doubt, contact the Data Engineering team.
Reset my Kerberos password?
File a task with the "Data Engineering" tag and we'll reset it for you.
Run a command as a system user?
See Data Platform/Systems/System users
Use JDBC with Hive Server
Use the following connection string (adding custom parameter that you need of course):
jdbc:hive2://analytics-hive.eqiad.wmnet:10000/default;principal=hive/analytics-hive.eqiad.wmnet@WIKIMEDIA
The principal=hive/analytics-hive.eqiad.wmnet@WIKIMEDIA part may look weird at first, since we'd expect to put our credentials in there. In JDBC it seems that you need to provide the identity of the target Kerberos principal, not yours (that will be automatically picked up from the credentials cache) to instruct Hive to use Kerberos. See Hive docs for more info.
Use Spark 2
On stat100[4,5,7] and notebook100[3,4] authenticate via kinit and then use the spark shell as you are used to. There are currently some limitations:
spark2-thriftserverrequires the hive keytab, that is only present on an-coord1001, so when running on client nodes it will return the following error:org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab
Use Jupyterhub (SWAP replica)
You can authenticate to Kerberos running kinit in the Terminal window. Please remember that it will be needed only once every 24h, not every time.
Administration
Check the status of the HDFS Namenodes and Yarn Resource Managers
Most of the commands are the same, but of course to authenticate as the user hdfs you'll need to use a keytab:
sudo -u hdfs kerberos-run-command hdfs /usr/bin/yarn rmadmin -getServiceState an-master1001-eqiad-wmnet
sudo -u hdfs kerberos-run-command hdfs /usr/bin/hdfs haadmin -getServiceState an-master1002-eqiad-wmnet