SRE/Dc-operations/Platform-specific documentation/Atlas Anchor

SRE Data Center Operations

DC Operations | About | Projects & Workboards | IRC: #wikimedia-dcops ^connect
HW Troubleshooting | HW Specific Documentation

Wikimedia infrastructure

Data centers

• Data center locations:
  • eqiad data center (Ashburn, USA)
  • codfw data center (Dallas, USA)
  • esams data center (Amsterdam, NL)
  • ulsfo data center (San Francisco, USA)
  • eqsin data center (Singapore, SG)
  • drmrs data center (Marseille, FR)
  • magru data center (São Paulo, BR)
• Data center operations
  • Common data center spec
  • Platform-specific doc
• Naming conventions

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

SRE Observability

Wikimedia Performance

Event Platform

Release Engineering

[]

Initial Setup Directions

NOTE: Instructions relate to the RIPE Atlas Anchor v3 (PC Engines APU.2E2 Hardware), which is the latest version at time of writing.

1. Connect port eth0 to switch port configured in Netbox (Cat-5E / copper link, 1000Base-T switch or copper SFP).
2. Connect DB9 Console port to OpenGear SCS port defined in Netbox (DB9 -> RJ45 cable).

Firmware Image

NOTE: Examples on this page are for Linux based systems, and may need adjusting in other environments.

1. Get Device Image from RIPE (netops)

Every Atlas anchor requires a device-specific firmware image prepared for it by the RIPE NCC. These images are pre-built with all IP addressing required, which we must supply to RIPE when requesting to host a new Anchor (or replacing an existing one).

• Netops can take care of this task.
• Requests for new or replacement anchors can be done via https://atlas.ripe.net/anchors/apply/
• RIPE supply download link after request is approved via email.

2. Download Device Image to Local Machine

Downloaded firmware images should be placed in /srv/firmware/ on apt1001.wikimedia.org. You should SSH to this machine and list the files in that directory to identify the name of the required image:

cmooney@apt1001:~$ ls -lahtr /srv/firmware/
total 8.6G
-rw-r--r--  1 root    root       0 Feb 19  2014 index.html
-rw-r--r--  1 root    root     41M Jul 13  2016 SAS-RAID_Firmware_4CGCG_LN_25.4.1.0004_A07.BIN
-rw-r--r--  1 root    root    115M Nov 10  2016 iDRAC-with-Lifecycle-Controller_Firmware_XTPX4_LN_2.41.40.40_A00.BIN
-rw-r--r--  1 root    root    4.7G Aug 31  2017 HP.SPP.881936_001_spp-2017.07.1-SPP2017071.2017_0718.11.iso
-rwxr-xr-x  1 robh    wikidev  17M Aug 31  2017 hp.ilo4_254.bin
-rw-r--r--  1 faidon  wikidev 954M Oct 26  2017 anchor.sg-sin-as14907.img
-rw-r--r--  1 faidon  wikidev   92 Oct 26  2017 anchor.sg-sin-as14907.img.sha256
-rw-r--r--  1 root    root    954M Nov  7  2017 anchor.nl-ams-as14907.img
-rw-r--r--  1 root    root      92 Nov  7  2017 anchor.nl-ams-as14907.img.sha256
-rw-r--r--  1 root    root    6.0M Jan 10  2018 firmware-smartarray-ea3138d8e8-6.06.tar.gz
-rw-r--r--  1 filippo wikidev 6.1M Apr 23  2018 firmware-smartarray-ea3138d8e8-6.30.tgz
-rw-r--r--  1 root    root    6.1M Jan 16  2019 firmware-smartarray-ea3138d8e8-6.60-1.1.x86_64.tgz
-rw-r--r--  1 root    root    6.2M Jul  3  2019 firmware-smartarray-ea3138d8e8-6.88-1.1.x86_64.tgz
-rw-r--r--  1 root    root      64 Nov 21  2019 anchor.nl-ams-as14907-v2.img.sha256
-rw-r--r--  1 root    root    954M Nov 21  2019 anchor.nl-ams-as14907-v2.img
drwxr-xr-x 12 root    root    4.0K Apr  3  2020 ..
-rw-r--r--  1 root    root      64 Aug 17 10:00 anchor.us-dal-as14907-v3.img.sha256
-rw-r--r--  1 root    root    954M Aug 17 10:00 anchor.us-dal-as14907-v3.img
drwxr-xr-x  2 root    root    4.0K Aug 17 13:06 .
cmooney@apt1001:~$ 

The image should be downloaded to your local laptop using scp or other SSH-based mechanism (sftp, rsync etc). For instance if the file we want is "anchor.us-dal-as14907-v3.img" then run this command

cmooney@wikilap:~$ scp cmooney@apt1001.wikimedia.org:/srv/firmware/anchor.us-dal-as14907-v3.img .
anchor.us-dal-as14907-v3.img                                         0% 3456KB 250.0KB/s 1:04:52 ETA

When complete display the saved hash value on apt1001, filename is the same with ".sha256" added:

cmooney@apt1001:~$ more /srv/firmware/anchor.us-dal-as14907-v3.img.sha256
e4689be071861a0557bde92acb48ec81aad20305684d17995a45b060a3dc80b2
cmooney@apt1001:~$ 

The value should match the hash calculated from the downloaded image:

cmooney@wikilap:~$ sha256sum anchor.us-dal-as14907-v3.img
e4689be071861a0557bde92acb48ec81aad20305684d17995a45b060a3dc80b2  anchor.us-dal-as14907-v3.img

3. Prepare USB Drive

This step requires a USB drive, larger than the image file itself (2GB+ should be ok). Note all data on the drive will be erased by the operation!

1. Identify USB device

After inserting the disk to your machine you need to identify the Linux device name that has been assigned to it. There are multiple ways to do this, one good way to start is to use 'lsblk':

cmooney@wikilap:~$ sudo lsblk | grep disk
sdb           8:0    1   3.7G  0 disk 
nvme0n1     259:0    0   477G  0 disk 

In my case above the USB drive is "sdb". It's fairly easy to identify because of the size. Be very careful on this point however, if in any doubt double-check or seek support, as the next operation will erase the contents of the device.

2. Unmount filesystems if needed

Unmount any filesystems on the USB drive that might be in use / have auto-mounted:

cmooney@wikilap:~$ sudo umount -v /dev/sdb*
umount: /dev/sdb: not mounted.
umount: /media/cmooney/c5b4171c-85d2-4dd3-ae55-96d37677e7b0 (/dev/sdb1) unmounted

3. Write image

To write the image to the identified disk we can use the 'dd' command:

cmooney@wikilap:~$ sudo dd if=anchor.us-dal-as14907-v3.img of=/dev/sda status=progress
997302784 bytes (997 MB, 951 MiB) copied, 305 s, 3.3 MB/s
1953125+0 records in
1953125+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 313.572 s, 3.2 MB/s
cmooney@wikilap:~$ 
cmooney@wikilap:~$ sync
cmooney@wikilap:~$

That should be it for creating the USB drive, remove from the computer.

Boot From USB to (re)Install Anchor

Installing the Anchor OS should be as easy as inserting the prepared USB device in the Anchor unit (while powered off), then restoring power to the Anchor.

The console output should be monitored during this process (netops?) to validate things are going ok, but it is a fully automatic process. The process beings by booting a CentOS Linux from the USB key, after which it installs the Anchor software to the device. The install looks as follows on the console:

Starting installer, one moment...
anaconda 21.48.22.93-1 for CentOS Linux 7 started.
 * installation log files are stored in /tmp    
                           ....
Installing nss-softokn (62/299)
Installing libnl3-cli (63/299)
Installing libassuan (64/299)
<--- output cut --->
Installing iwl135-firmware (295/299)
Installing iwl1000-firmware (297/299)
Performing post-installation setup tasks

No further output was seen on the console. Once 30 minutes and the last line above have been seen on the console the USB drive should be removed from the device, and it should be power cycled again, after which is should boot from it's internal drive to an online state.

Validate System is Online

Netops to complete these steps.

When the above install process is finished the associated switch port should show up/up with a MAC address learnt on it. Device allocated IPs should be pingable.

Provided this is the case log on to the RIPE NCC portal (https://atlas.ripe.net/anchors/manage/), select 'Anchors' and click the 'Install Software' button next to the newly installed instance.

Following that RIPE will perform various checks, and email us to confirm the status. After that it should be publicly visible (at https://atlas.ripe.net/anchors/list/).