Jump to content

User:Razzi/First logical volume resizing

From Wikitech
puppet $ sshrc an-test-coord1001.eqiad.wmnet
Linux an-test-coord1001 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
Debian GNU/Linux 10 (buster)

  _  __         _               _             _   _               _
 | |/ /        | |             (_)           | | | |             | |
 | ' / ___ _ __| |__   ___ _ __ _ _______  __| | | |__   ___  ___| |_
 |  < / _ \ '__| '_ \ / _ \ '__| |_  / _ \/ _` | | '_ \ / _ \/ __| __|
 | . \  __/ |  | |_) |  __/ |  | |/ /  __/ (_| | | | | | (_) \__ \ |_
 |_|\_\___|_|  |_.__/ \___|_|  |_/___\___|\__,_| |_| |_|\___/|___/\__|


This host is capable of Kerberos authentication in the WIKIMEDIA realm.

For more info: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide

an-test-coord1001 is a Analytics Cluster host running various Hadoop services (Hive, Camus, Oozie, ..) and maintenance scripts (analytics_test_cluster::coordinator)
Debian GNU/Linux 10 auto-installed on Fri Apr 2 14:23:58 UTC 2021.
Last login: Mon Jan 24 16:57:47 2022 from 2620:0:863:1:198

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.
welcome
razzi@an-test-coord1001:~$ ls
apache-atlas-2.2.0-sources.tar  cookiejar.txt  pidinfo.txt
apache-atlas-sources-2.2.0      index.html     run_atlas
razzi@an-test-coord1001:~$ df -h
df: /mnt/hdfs: Input/output error
Filesystem            Size  Used Avail Use% Mounted on
udev                   63G     0   63G   0% /dev
tmpfs                  13G  1.4G   12G  11% /run
/dev/mapper/vg0-root   73G   50G   20G  72% /
tmpfs                  63G     0   63G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/vg0-srv   365G   29G  318G   9% /srv
tmpfs                  13G     0   13G   0% /run/user/0
tmpfs                  13G     0   13G   0% /run/user/118
tmpfs                  13G     0   13G   0% /run/user/32741
tmpfs                  13G     0   13G   0% /run/user/2543
tmpfs                  13G     0   13G   0% /run/user/26051
razzi@an-test-coord1001:~$ man lvreduce
razzi@an-test-coord1001:~$ sudo lvs
  LV   VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root vg0 -wi-ao----   74.50g
  srv  vg0 -wi-ao---- <371.27g
  swap vg0 -wi-ao----  976.00m
razzi@an-test-coord1001:~$ lvreduce srv -L -2
bash: lvreduce: command not found
razzi@an-test-coord1001:~$ sudo lvreduce srv -L -2
  Please specify a logical volume path.
  Run `lvreduce --help' for more information.
razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-srv -L -2G
  WARNING: Reducing active and open logical volume to <369.27 GiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce vg0/srv? [y/n]: y
  Size of logical volume vg0/srv changed from <371.27 GiB (95044 extents) to <369.27 GiB (94532 extents).
  Logical volume vg0/srv successfully resized.
razzi@an-test-coord1001:~$ sudo lvreduce /dev/mapper/vg0-root -L +2G
  Size may not be positive.
  Invalid argument for --size: +2G
  Error during parsing of command line.
razzi@an-test-coord1001:~$ sudo lvextend /dev/mapper/vg0-root -L+2G
  Size of logical volume vg0/root changed from 74.50 GiB (19073 extents) to 76.50 GiB (19585 extents).
  Logical volume vg0/root successfully resized.
razzi@an-test-coord1001:~$ resize2fs /dev/mapper/vg0-root
bash: resize2fs: command not found
razzi@an-test-coord1001:~$ sudo resize2fs /dev/mapper/vg0-root
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/mapper/vg0-root is mounted on /; on-line resizing required
old_desc_blocks = 10, new_desc_blocks = 10
The filesystem on /dev/mapper/vg0-root is now 20055040 (4k) blocks long.

razzi@an-test-coord1001:~$ df -h
df: /mnt/hdfs: Input/output error
Filesystem            Size  Used Avail Use% Mounted on
udev                   63G     0   63G   0% /dev
tmpfs                  13G  1.4G   12G  11% /run
/dev/mapper/vg0-root   75G   50G   22G  70% /
tmpfs                  63G     0   63G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  63G     0   63G   0% /sys/fs/cgroup
/dev/mapper/vg0-srv   365G   29G  318G   9% /srv
tmpfs                  13G     0   13G   0% /run/user/0
tmpfs                  13G     0   13G   0% /run/user/118
tmpfs                  13G     0   13G   0% /run/user/32741
tmpfs                  13G     0   13G   0% /run/user/2543
tmpfs                  13G     0   13G   0% /run/user/26051

I did the above in the wrong order; we were fortunately able to create a backup of mariadb from the corrupted volume and restore that to srv. Some other folders were manually recreated and some were created from a puppet run. The puppet run errored however, with a systemd issue:

Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ...
 systemd[1]: Caught <ABRT>, dumped core as pid 15168.

Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC):

systemd[1]: Caught <ABRT>, dumped core as pid 15168.


Message from syslogd@an-test-coord1001 at Jan 24 20:57:01 ...
 systemd[1]: Freezing execution.

Broadcast message from systemd-journald@an-test-coord1001 (Mon 2022-01-24 20:57:01 UTC):

systemd[1]: Freezing execution.


razzi@an-test-coord1001:~$