Howto solve graylog indexer failures due no more diskspace, without reboot?

edited August 13 in Graylog

Parts of this faq are based on an Ubuntu server install. For other distri, some commands may be different.
Also I assume you have the required root or sudo rights and know how to use this.
Of course: "BACKUP BEFORE YOU START" is the best practise!!

Incoming messages in input:

system-overview show indexer failures

Log in to your server by ssh (or console)
$ > sudo -i

The output will be something like this:

The command 'df -h' will show a little to no more disk space for your partition assigned as data partition (often / partition)

Solution:

First expand disk @ hypervisor level
To check which disks are seen, execute the command below
:~# ls /sys/class/scsi_device/

Output (example):

0:0:0:0 1:0:0:0 2:0:0:0

Then rescan the scsi bus. Below you can replace the ‘0:0:0:0’ with the actual scsi bus name found with the previous command.
Each colon is prefixed with a slash, which is what makes it look weird.

:~# echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/rescan

Then to fix disk cylindernumbers and size to actual (new) size:

:~# parted /dev/sda unit s print
Output:

Warning: Not all of the space available to /dev/sda appears to be used, you can fix the GPT to use all of the space (an extra 524288000 blocks) or continue with the current setting?
Fix/Ignore? fix
Model: Msft Virtual Disk (scsi)
Disk /dev/sda: 1048576000s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 2048s 1050623s 1048576s fat32 boot, esp
2 1050624s 3147775s 2097152s ext4
3 3147776s 524285951s 521138176s

Grow your datapartition to max size. In my case partition 3

Dryrun:
:~# growpart --dry-run /dev/sda 3

Actual adjustment:
:~# growpart /dev/sda 3

Check if partitions are correctly adjusted:
:~# fdisk -l /dev/sda
Output:

Disk /dev/sda: 500 GiB, 536870912000 bytes, 1048576000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9BBD622E-C14D-4B25-8078-19426D70E449
Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 3147775 2097152 1G Linux filesystem
/dev/sda3 3147776 1048575966 1045428191 498.5G Linux filesystem

Now we can extend the LVM
Since I use webmin for all linux machines, here I step over to Webmin console, but I post here the "How to" for those who wish to use cli further:
Increase VMware disk size
How to extend LVM volume


When the LVM has grown to the max sizes as required, we can enable graylog again to write indexes
:~# curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Sign In or Register to comment.