Why?


Search This Blog

Sunday, March 12, 2017

Bad luck with WD Red NAS 3TB drives

3/12/2017

I have had 4 of 8 drives go bad in the last year and half. Oldest drive is 2 years old. Model numbers for all the drives are WD30EFRX-68EUZN0. I am waiting on an RMA to come back, and another went bad tonight. Lucky I have extra drives and am replacing tonight with another new WD Red NAS 3TB drive i had for just this purpose.

I have decided to not get anymore of these drives, other than the RMA I am waiting on now, and the RMA I will do after I replace the current bad drive. I have ordered a single HGST Desktsar NAS 3TB drive to look at. If this will fit my needs I will replace all drives in my NAS with these.

I want to look at the heat, noise, and see if i can get head parking off on the HGST drives. I will also measure performance, but I am sure the 7200 HGST will be faster then the 5400 WD Red.

Just an FYI to everyone. Be careful with these WD Red NAS 3TB drives.

UPDATE 3/13/2017

So i got the drive out of the NAS. Home built  on Centos 7.3, kernel 4.10, btrfs-progs 4.9. Took a bit to get the drive removed from the array (wait for data to be moved across the other 7 drives) and then add the replacement back in (wait for data to be balanced back across all 8 drives again).

I then took the "bad drive" and put it in my Windows workstation. I ran the WD Data Lifeguard extended test and then ran the extended test from Aomei Partition Assistant Pro. Both of these test show I have no bad sectors. The drive seems to be good as new. To make long story short I think i have a bad fan cable from my SAS/SATA controller to the drives. I have two ports on controller that controls 4 SATA drives each. I reseated HBA card and cables, and power connectors on the drives, and ran some test and scrubs. all seems good with new drive. I then put the side panels back on the case and place it back in its home. And within an hour i start getting more errors on the same device id, only with the brand new drive on that id. So i pull the server back out, remove side panels, and reboot. Over 24 hours later I don't have another error. It maybe that the side panels on the case are moving the cables enough for the bad cable to cause the errors. So i have two new fan cables coming and should have them installed tomorrow night. I still have the new HGST drive coming because I really want to check this out. If they are not to noisy and run to hot, I may just replace them all anyway. I better see some latency and IO improvements before that happens. At $150.00 per drive x8..... Well its not cheap to me.

Friday, March 3, 2017

BTRFS Drive errors from 2017-02-26

BTRFS Drive errors from 2017-02-26

Was not getting failed scrubs but seeing some errors.

This is 8x 3TB WD Red drives in raid10 on a Supermicro AOC-SAS2LP-MV8 Add-on Card, 8-Channel SAS/SATA Adapter with 600MB/s per Channel in a PCIE x16 slot running at x8 on a Supermicro ATX DDR4 LGA 1151 C7Z170-OCE-O Motherboard with 64GB DDR4 RAM (4x 16GB sticks).

FYI Everything was done online unless otherwise stated.

What file system looks like in btrfs.

[root@nas ~]# btrfs fi show
Label: 'myraid'  uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 8 FS bytes used 1.16TiB
        devid    1 size 2.73TiB used 301.53GiB path /dev/sdb
        devid    2 size 2.73TiB used 301.53GiB path /dev/sdc
        devid    3 size 2.73TiB used 301.53GiB path /dev/sdd
        devid    4 size 2.73TiB used 301.53GiB path /dev/sde
        devid    6 size 2.73TiB used 301.53GiB path /dev/sdg
        devid    7 size 2.73TiB used 301.53GiB path /dev/sdh
        devid    8 size 2.73TiB used 301.53GiB path /dev/sdi
        devid    9 size 2.73TiB used 301.53GiB path /dev/sdf


Take a look at device stats.

[root@nas ~]# /usr/local/bin/btrfs device stats /myraid/
[/dev/sdb].write_io_errs   0
[/dev/sdb].read_io_errs    0
[/dev/sdb].flush_io_errs   0
[/dev/sdb].corruption_errs 0
[/dev/sdb].generation_errs 0
[/dev/sdc].write_io_errs   0
[/dev/sdc].read_io_errs    0
[/dev/sdc].flush_io_errs   0
[/dev/sdc].corruption_errs 0
[/dev/sdc].generation_errs 0
[/dev/sdd].write_io_errs   0
[/dev/sdd].read_io_errs    0
[/dev/sdd].flush_io_errs   0
[/dev/sdd].corruption_errs 0
[/dev/sdd].generation_errs 0
[/dev/sde].write_io_errs   0
[/dev/sde].read_io_errs    44
[/dev/sde].flush_io_errs   0
[/dev/sde].corruption_errs 0
[/dev/sde].generation_errs 0
[/dev/sdg].write_io_errs   0
[/dev/sdg].read_io_errs    0
[/dev/sdg].flush_io_errs   0
[/dev/sdg].corruption_errs 0
[/dev/sdg].generation_errs 0
[/dev/sdh].write_io_errs   0
[/dev/sdh].read_io_errs    0
[/dev/sdh].flush_io_errs   0
[/dev/sdh].corruption_errs 0
[/dev/sdh].generation_errs 0
[/dev/sdi].write_io_errs   0
[/dev/sdi].read_io_errs    0
[/dev/sdi].flush_io_errs   0
[/dev/sdi].corruption_errs 0
[/dev/sdi].generation_errs 0
[/dev/sdf].write_io_errs   0
[/dev/sdf].read_io_errs    0
[/dev/sdf].flush_io_errs   0
[/dev/sdf].corruption_errs 0
[/dev/sdf].generation_errs 0


Run extended smartcl test.

[root@nas ~]# smartctl -t long /dev/sdb
[root@nas ~]# smartctl -t long /dev/sdc
[root@nas ~]# smartctl -t long /dev/sdd
[root@nas ~]# smartctl -t long /dev/sde
[root@nas ~]# smartctl -t long /dev/sdf
[root@nas ~]# smartctl -t long /dev/sdg
[root@nas ~]# smartctl -t long /dev/sdh
[root@nas ~]# smartctl -t long /dev/sdi


I waited an hour then reviewed the results.

[root@nas ~]# smartctl -l selftest /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12660         -
# 2  Extended offline    Completed without error       00%      8916         -
# 3  Short offline       Completed without error       00%      6097         -
# 4  Extended offline    Completed without error       00%      4288         -
# 5  Short offline       Completed without error       00%      4245         -
# 6  Short offline       Completed without error       00%      4242         -
# 7  Short offline       Interrupted (host reset)      50%      4241         -
# 8  Short offline       Completed without error       00%      4172         -
# 9  Short offline       Completed without error       00%      4109         -

[root@nas ~]# smartctl -l selftest /dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12660         -
# 2  Extended offline    Completed without error       00%      8916         -
# 3  Short offline       Completed without error       00%      6096         -
# 4  Extended offline    Completed without error       00%      4288         -
# 5  Short offline       Completed without error       00%      4109         -

[root@nas ~]# smartctl -l selftest /dev/sdd
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     13003         -
# 2  Extended offline    Completed without error       00%      9260         -
# 3  Short offline       Completed without error       00%      6440         -
# 4  Extended offline    Completed without error       00%      4632         -
# 5  Short offline       Completed without error       00%      4452         -
# 6  Short offline       Completed without error       00%         0         -

[root@nas ~]# smartctl -l selftest /dev/sde
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      8452         339784376
# 2  Extended offline    Completed without error       00%      4716         -
# 3  Short offline       Completed without error       00%      1896         -
# 4  Extended offline    Completed without error       00%        88         -
# 5  Short offline       Completed without error       00%        15         -
# 6  Short offline       Aborted by host               90%        12         -
# 7  Short offline       Aborted by host               90%        12         -
# 8  Short offline       Aborted by host               90%        12         -
# 9  Short offline       Aborted by host               90%         5         -
#10  Short offline       Aborted by host               90%         5         -
#11  Short offline       Aborted by host               90%         5         -
#12  Short offline       Aborted by host               90%         4         -
#13  Short offline       Aborted by host               90%         4         -
#14  Short offline       Aborted by host               90%         4         -
#15  Short offline       Aborted by host               90%         0         -
#16  Short offline       Aborted by host               90%         0         -

[root@nas ~]# smartctl -l selftest /dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      3118         -
# 2  Short offline       Completed without error       00%        12         -

[root@nas ~]# smartctl -l selftest /dev/sdg
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6298         -
# 2  Extended offline    Completed without error       00%      2555         -

[root@nas ~]# smartctl -l selftest /dev/sdh
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6147         -
# 2  Extended offline    Completed without error       00%      2404         -
# 3  Short offline       Completed without error       00%         0         -

[root@nas ~]# smartctl -l selftest /dev/sdi
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6147         -
# 2  Extended offline    Completed without error       00%      2404         -
# 3  Short offline       Completed without error       00%         0         -



[root@nas ~]# smartctl -a /dev/sdb | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdc | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdd | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sde | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1258
[root@nas ~]# smartctl -a /dev/sdf | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdg | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdh | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdi | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

 

Yup. /dev/sde seems to have an issue.



Get serial number of all drives for a possible RMA on /dev/sde.

[root@nas ~]# smartctl -a /dev/sdb | grep "Serial Number:"
Serial Number:    WD-WMC4N0J0YT1V
[root@nas ~]# smartctl -a /dev/sdc | grep "Serial Number:"
Serial Number:    WD-WMC4N0J2L138
[root@nas ~]# smartctl -a /dev/sdd | grep "Serial Number:"
Serial Number:    WD-WCC4N2FJRTU9
[root@nas ~]# smartctl -a /dev/sde | grep "Serial Number:"
Serial Number:    WD-WCC4N4SSDRFN
[root@nas ~]# smartctl -a /dev/sdf | grep "Serial Number:"
Serial Number:    WD-WCC4N1VYZH52
[root@nas ~]# smartctl -a /dev/sdg | grep "Serial Number:"
Serial Number:    WD-WMC4N0M57KEY
[root@nas ~]# smartctl -a /dev/sdh | grep "Serial Number:"
Serial Number:    WD-WCC4N5YF2Z2Y
[root@nas ~]# smartctl -a /dev/sdi | grep "Serial Number:"
Serial Number:    WD-WCC4N5CJ6H8U




Get List of BadBlocks on all drives. Run in background and save to file.

badblocks -v /dev/sdb > /tmp/bad-blocks-b.txt &
badblocks -v /dev/sdc > /tmp/bad-blocks-c.txt &
badblocks -v /dev/sdd > /tmp/bad-blocks-d.txt &
badblocks -v /dev/sde > /tmp/bad-blocks-e.txt &
badblocks -v /dev/sdf > /tmp/bad-blocks-f.txt &
badblocks -v /dev/sdg > /tmp/bad-blocks-g.txt &
badblocks -v /dev/sdh > /tmp/bad-blocks-h.txt &
badblocks -v /dev/sdi > /tmp/bad-blocks-i.txt &

Monitor file size with:

[root@nas ~]# watch ls -lsa /tmp/bad-blocks-*.txt

If you have a really bad drive it could create a file the size of the drive itself so be sure to monitor and make sure you do not fill up your /tmp directory.

If you need to kill it then ket the pid with:

[root@nas tmp]# ps -ef | grep "badblocks"
UID        PID  PPID  C STIME TTY      TIME     CMD
root     27013 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdb
root     27014 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdc
root     27015 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdd
root     27016 25404  2 10:43 pts/0    00:01:11 badblocks -v /dev/sde
root     27017 25404  3 10:43 pts/0    00:01:13 badblocks -v /dev/sdf
root     27018 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdg
root     27019 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdh
root     27020 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdi
root     31044 26976  0 11:22 pts/1    00:00:00 grep --color=auto badblocks



While badblock test is running I have already got a RMA number from WD and a shipping label on my printer. I ordered a new drive from Amazon that will be here on the 28th. I will swap out then and ship bad drive back on the 29th.

Running smartctl test long on all drives

smartctl -t long /dev/sdb
smartctl -t long /dev/sdc
smartctl -t long /dev/sdd
smartctl -t long /dev/sde
smartctl -t long /dev/sdf
smartctl -t long /dev/sdg
smartctl -t long /dev/sdh
smartctl -t long /dev/sdi

Check progress of test

smartctl -a /dev/sdb | grep "Self-test execution status"
smartctl -a /dev/sdb | grep "of test remaining."

smartctl -a /dev/sdc | grep "Self-test execution status"
smartctl -a /dev/sdc | grep "of test remaining."

smartctl -a /dev/sdd | grep "Self-test execution status"
smartctl -a /dev/sdd | grep "of test remaining."

smartctl -a /dev/sde | grep "Self-test execution status"
smartctl -a /dev/sde | grep "of test remaining."

smartctl -a /dev/sdf | grep "Self-test execution status"
smartctl -a /dev/sdf | grep "of test remaining."

smartctl -a /dev/sdg | grep "Self-test execution status"
smartctl -a /dev/sdg | grep "of test remaining."

smartctl -a /dev/sdh | grep "Self-test execution status"
smartctl -a /dev/sdh | grep "of test remaining."

smartct
l -a /dev/sdi | grep "Self-test execution status"
smartctl -a /dev/sdi | grep "of test remaining."



New drive is in and I am backing up NAS. My btrfs pool of /myraid is gettin backuped up to a PC with raid1.
I am also duplicating the more inportant files to a SSD. Better safe than sorry.

New drive has had a full surface test (9 hours) and passed with flying colors.

I am not turning this into a drive remove/replace since i want to change the partions of my boot SSD so I am just
goging to nuke the entire system and rebuild from scratch.

my /(root) partion is getting a backup vi tar so i can have access to my old crontab files and maintemce scripts.
I can just yse the old samba.conf as well, etc....



tar -zcvpf /myraid/nas.backup.tar.gz --exclude=/myraid --exclude=/usr --exclude=/proc --exclude=/lib --exclude=/lib64 --exclude=/dev /

Now copy the tar.gz file to a few drives off the server as well.

Make USB install Centos 7.3 min and do the install :)

Install done. I see all 9 drives and 4 network connections. I setup all the NICs during the install and they all seem to be ok
Possibily some tweeking on these later.

I installed the OS on my SSD with 1GB /boot and /boot/efi (I am using EFI). The rest to /

My other 8 drives are on my Supermicro AOC-SAS2LP-MV8 JBOD HBA. I will not touch those untill I get ready to setup btrfs on them.

So now some base stuff


cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux

cp /etc/selinux/config /etc/selinux/config.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

systemctl disable firewalld
systemctl stop firewalld

service iptables stop
service ip6tables stop
chkconfig iptables off
chkconfig ip6tables off

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git vim gdisk parted

yum -y groupinstall "Development Tools"
yum -y update
yum -y upgrade

cd /root
echo ':color desert' > .vimrc

systemctl disable kdump.service

reboot

# cat /etc/default/grub
GRUB_TIMEOUT=60
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl_bcache/root ipv6.disable=1 zswap.enable=1 consoleblank=0"
GRUB_DISABLE_RECOVERY="true"

Make changes as refelected about to
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl_bcache/root ipv6.disable=1 zswap.enable=1 consoleblank=0"

# grub2-mkconfig -o /boot/grub2/grub.cfg
or if using UEFI
# grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

reboot


Now update kerenel

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install yum-plugin-fastestmirror

yum --enablerepo=elrepo-kernel install kernel-ml

reboot

Manualy select new kernel from grub boot screen.

uname -r
4.10.1-1.el7.elrepo.x86_64

Do any testing you need and when happy set this to default entry when happy.

grub2-set-default 0

reboot

uname -r
4.10.1-1.el7.elrepo.x86_64


Now time to physicaly replace drive then setup btrfs.

poweroff

New drive in and server rebooted

parted -l

shows all drives. 7x WD RED NAS drives show still brtfs partitions. New drive has nothing.

idle3ctl show new drive has head parking on. lets turn that off.

# ./idle3ctl /dev/sde
Idle3 timer set to 138 (0x8a)

# ./idle3ctl -d /dev/sde
Idle3 timer disabled
Please power cycle your drive off and on for the new setting to be taken into account. A reboot will not be enough!

So lets powerr off, let set for a min, and power back on and rechedck.

poweroff

looks good on all drives 8 WD RED drives

[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdb
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdc
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdd
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sde
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdf
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdg
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdh
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdi
Idle3 timer is disabled


No lets clean the disk for a new array

I used parted to rm the partiions then a w and q.

Then

wipefs -a /dev/sdb
wipefs -a /dev/sdc
wipefs -a /dev/sdd
wipefs -a /dev/sde
wipefs -a /dev/sdf
wipefs -a /dev/sdg
wipefs -a /dev/sdh
wipefs -a /dev/sdi

Then

dd if=/dev/zero of=/dev/sdb bs=1024 count=1024
dd if=/dev/zero of=/dev/sdc bs=1024 count=1024
dd if=/dev/zero of=/dev/sdd bs=1024 count=1024
dd if=/dev/zero of=/dev/sde bs=1024 count=1024
dd if=/dev/zero of=/dev/sdf bs=1024 count=1024
dd if=/dev/zero of=/dev/sdg bs=1024 count=1024
dd if=/dev/zero of=/dev/sdh bs=1024 count=1024
dd if=/dev/zero of=/dev/sdi bs=1024 count=1024

Then to just look at the devs

ls -lsa /dev/sd*
0 brw-rw---- 1 root disk 8,   0 Mar  1 15:02 /dev/sda
0 brw-rw---- 1 root disk 8,   1 Mar  1 15:02 /dev/sda1
0 brw-rw---- 1 root disk 8,   2 Mar  1 15:02 /dev/sda2
0 brw-rw---- 1 root disk 8,   3 Mar  1 15:02 /dev/sda3
0 brw-rw---- 1 root disk 8,  16 Mar  1 15:11 /dev/sdb
0 brw-rw---- 1 root disk 8,  32 Mar  1 15:11 /dev/sdc
0 brw-rw---- 1 root disk 8,  48 Mar  1 15:11 /dev/sdd
0 brw-rw---- 1 root disk 8,  64 Mar  1 15:11 /dev/sde
0 brw-rw---- 1 root disk 8,  80 Mar  1 15:11 /dev/sdf
0 brw-rw---- 1 root disk 8,  96 Mar  1 15:11 /dev/sdg
0 brw-rw---- 1 root disk 8, 112 Mar  1 15:11 /dev/sdh
0 brw-rw---- 1 root disk 8, 128 Mar  1 15:11 /dev/sdi



fdisk -l also shows they look ready


[root@nas idle3-tools-0.9.1]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048      2099199      1G  EFI System      EFI System Partition
 2      2099200      4196351      1G  Microsoft basic
 3      4196352    250068991  117.2G  Linux LVM      

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdf: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdg: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdh: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdi: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/cl_nas-root: 125.9 GB, 125883645952 bytes, 245866496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes



Check out btrfs version first

# btrfs --version
btrfs-progs v4.4.1

I think there is a newer on out. Lets go see.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

cd btrfs-progs

yum -y install libuuid-devel libattr-devel zlib-devel libacl-devel e2fsprogs-devel libblkid-devel lzo* asciidoc xmlto

./autogen.sh
./configure
make

Lets check version from within the folder

[root@nas btrfs-progs]# ./btrfs --version
btrfs-progs v4.9.1

Yup its newer

Now check from /

[root@nas btrfs-progs]# cd /
[root@nas /]# btrfs --version
btrfs-progs v4.4.1
[root@nas /]#

So we got two versions

I copied all +x files from /root/btrfs-progs to /usr/sbin overright files if they exist.

Now from / of drive I get

[root@nas /]# btrfs --version
btrfs-progs v4.9.1

I hope thats good :)

So lets build an array!!!

First I will use raid0 for some quick testing.

mkfs.btrfs -f -m raid0 -d raid0 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

also here is for raid10

mkfs.btrfs -f -m raid10 -d raid10 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

[root@nas ~]# mkfs.btrfs -f -m raid0 -d raid0 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi
btrfs-progs v4.9.1
See http://btrfs.wiki.kernel.org for more information.

Label:              myraid
UUID:               5a5610aa-2615-4ee2-bd4a-076ab2931b70
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
  Data:             RAID0             8.00GiB
  Metadata:         RAID0             4.00GiB
  System:           RAID0            16.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  8
Devices:
   ID        SIZE  PATH
    1     2.73TiB  /dev/sdb
    2     2.73TiB  /dev/sdc
    3     2.73TiB  /dev/sdd
    4     2.73TiB  /dev/sde
    5     2.73TiB  /dev/sdf
    6     2.73TiB  /dev/sdg
    7     2.73TiB  /dev/sdh
    8     2.73TiB  /dev/sdi
   
[root@nas ~]# btrfs fi show
Label: 'myraid'  uuid: 5a5610aa-2615-4ee2-bd4a-076ab2931b70
        Total devices 8 FS bytes used 112.00KiB
        devid    1 size 2.73TiB used 1.50GiB path /dev/sdb
        devid    2 size 2.73TiB used 1.50GiB path /dev/sdc
        devid    3 size 2.73TiB used 1.50GiB path /dev/sdd
        devid    4 size 2.73TiB used 1.50GiB path /dev/sde
        devid    5 size 2.73TiB used 1.50GiB path /dev/sdf
        devid    6 size 2.73TiB used 1.50GiB path /dev/sdg
        devid    7 size 2.73TiB used 1.50GiB path /dev/sdh
        devid    8 size 2.73TiB used 1.50GiB path /dev/sdi

Lets mount this thing

mkdir /myraid

mount with UUID from above. uuid: 5a5610aa-2615-4ee2-bd4a-076ab2931b70

mount -t btrfs -o defaults,nodatacow,noatime,x-systemd.device-timeout=30 -U 5a5610aa-2615-4ee2-bd4a-076ab2931b70 /myraid

[root@nas ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                  32G     0   32G   0% /dev
tmpfs                     32G     0   32G   0% /dev/shm
tmpfs                     32G  8.9M   32G   1% /run
tmpfs                     32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/cl_nas-root  118G  2.2G  116G   2% /
/dev/sda2               1014M  191M  824M  19% /boot
/dev/sda1               1022M  9.5M 1013M   1% /boot/efi
tmpfs                    6.3G     0  6.3G   0% /run/user/0
/dev/sdb                  22T   20M   22T   1% /myraid

Oh ya!!! 22TB of btrfs array

My line for fstab i will put in later is:

UUID=5a5610aa-2615-4ee2-bd4a-076ab2931b70   /myraid   btrfs  defaults,nodatacow,noatime,x-systemd.device-timeout=30  0 0


this is how to clear cache when testing transfer speeds to make sure you are not using cache.
Do this between each transfer


sync; echo 2 > /proc/sys/vm/drop_caches
sync; echo 3 > /proc/sys/vm/drop_caches

---tune 10Gb CNA if needed

service irqbalance stop
service cpuspeed stop
chkconfig irqbalance off
chkconfig cpuspeed off
systemctl disable irqbalance
systemctl disable cpuspeed
systemctl stop irqbalance
systemctl stop cpuspeed

vi /etc/sysconfig/network-scripts/ifcfg-eth???
MTU="9000"

vi /etc/sysctl.conf
# -- tuning -- #
# Increase system file descriptor limit
fs.file-max = 65535

# Increase system IP port range to allow for more concurrent connections
net.ipv4.ip_local_port_range = 1024 65000

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

reboot and test speed.

on linux client pointing to server with ip 192.168.90.100

# iperf3 -c 192.168.90.100 -p 5201

on linux server with IP 192.168.90.100

iperf3 -s -p 5201 -B 192.168.90.100

---end tune 10Gb CNA if needed


---setup NFS for ESXi server

vi /etc/exports
/myraid/     192.168.10.0/24(rw,async,no_root_squash,no_subtree_check)
/myraid/     192.168.90.0/24(rw,async,no_root_squash,no_subtree_check)

systemctl start rpcbind nfs-server
systemctl enable rpcbind nfs-server

---end setup NFS for ESXi server




--install samaba if needed

yum -y install samba

useradd samba -s /sbin/nologin

smbpasswd -a samba
            Supply a password
            Retype the password
   
mkdir /myraid

chown -R samba:root /myraid/

vi /etc/samba/smb.conf

[global]
workgroup = WORKGROUP ;use name of your workgroup here
server string = Samba Server Version %v
netbios name = NAS

Add this to botton of /etc/samba/smb.conf file

[NAS]
comment = NAS
path = /myraid
writable = yes
valid users = samba


systemctl start smb
systemctl enable smb
systemctl start nmb
systemctl enable nmb

testparm
   
--end install samaba if needed



---install plex if needed


visit plex site and get rpm for your version of OS
copy this to /root

yum -y localinstall name.rpm

systemctl enable plexmediaserver
systemctl start plexmediaserver

---end install plex if needed



---install LAMP

yum -y install httpd mariadb-server mariadb php php-mysql
systemctl enable httpd.service
systemctl start httpd.service
systemctl status httpd.service

Make sure it works with:
http://your_server_IP_address/

systemctl enable mariadb
systemctl start mariadb
systemctl status mariadb
mysql_secure_installation

vi /var/www/html/info.php
<?php phpinfo(); ?>

http://your_server_IP_address/info.php


---End install LAMP


---Extra goodies

yum -y install epel-release
yum -y install stress htop iftop iotop hddtemp smartmontools iperf3 sysstat mlocate
yum -y update


updatedb **this is to update mlocate db


---End Extra goodies




---Use gmail as relay for sending mail

Replace glen@gmail.com with a real email address in items below
Replace mycentserver.mydomain.domain with real hostname in items below
Replace gmail_password with real password in items below

# yum remove postfix

Now install ssmtp.

# yum -y install ssmtp mailx

Now edit your  /etc/ssmtp/ssmtp.conf. I removed everything and just added the below in the file.

#  vi /etc/ssmtp/ssmtp.conf

root=glen@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=gmail.com
hostname=mycentserver.mydomain.domain
UseTLS=Yes
UseSTARTTLS=Yes
AuthUser=glen@gmail.com
AuthPass=gmail_password
FromLineOverride=YES

# This solved if you get a ssmtp: Cannot open smtp.gmail.com:587 when try to send an email
# if you enabled uncommenting DEBUG=Yes line and your /var/log/maillog show
# SSL not working: certificate verify failed (20) Uncomment the following line but first
# VERIFY FILE EXISTS
TLS_CA_File=/etc/pki/tls/certs/ca-bundle.crt

# DEBUG=Yes

Now edit your /etc/ssmtp/revaliases file and add the following.

# vi /etc/ssmtp/revaliases

root:glen@gmail.com:smtp.gmail.com:587

Now run

# alternatives --config mta

And choose the number for sendmail.ssmtp, like below

There is 1 program that provides 'mta'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/sbin/sendmail.ssmtp

Enter to keep the current selection[+], or type selection number: 1
#

Now send email to your gmail account from Centos cli

# mail -s "Test Subject" glen@gmail.com

Type your message text and on new line press ctrl d to send

---End Use gmail as relay for sending mail


Thursday, March 2, 2017

Centos 7 backing up my NAS

I put the tar in /myraid (my btrfs 8 drive raid10) then copy this off to another machine that then replicates to cloud.

So if my boot drive goes out I have the backup on the array, on another onsite machine, and in the cloud.

tar --exclude='/myraid' --exclude='/proc' --exclude='/sys' -czvf /myraid/nas_backup.tar.gz /

I don't have enough cloud storage to replicate /myraid but I copy the entire /myraid mount to another onsite machine with a raid1. I guess its better than nothing :)

Saturday, February 25, 2017

Centos 7 with two different networks

Centos 7 with two different networks

I have a vm image with two nics in it. One to my 10Gb 192.168.70.0 network. This network is not on a switch but simply a DAC to my PC. So my PC is 192.168.70.5 and my ESXi is 192.168.70.90.

I also have another network in the vm inage on my 1Gb 192.168.10.0 network. This is my may main network, with a switch, and a route out to internet (192.168.10.1). So my PC is 192.168.10.5, ESXi 192.168.10.90, and router 192.168.10.1.

IP addresses for the vm image itself are:
192.168.10.22
192.168.70.22


Problem is when I have the 192.168.70.22 enabled I am not able to reach the internet. To remedy this I have the folowing /etc/sysconfig/network-scripts ifcfg files:

# cat ifcfg-ens192
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=dcc2277d-a060-48a0-8829-350abf95e269
DEVICE=ens192
ONBOOT=yes
DNS1=192.168.10.1
IPADDR=192.168.70.22
PREFIX=24
#GATEWAY=192.168.70.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no


# cat ifcfg-ens224
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens224
UUID=78e2b2a8-d444-4fd2-9751-bb108861392a
DEVICE=ens224
ONBOOT=yes
IPADDR=192.168.10.22
PREFIX=24
GATEWAY=192.168.10.1
DEFROUTE=yes
DNS1=192.168.10.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no


I then change the value of rp_filter to loose mode, value 2 with:

# sysctl net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.all.rp_filter = 2

Restart network with:

# systemctl restart network

I can now ping internet (8.8.8.8), all nodes on 192.168.10.0, and all nodes on 192.168.70.0 (ESXi 192.168.70.90, PC 192.168.70.5 and the vm image of 192.168.70.22).

A quick reboot and re test of access and I am good.

I also want to tune the 10Gb interface as Im wanting to to test SSD disk caching on this image and need the fastest I can get. FYI My PC on 192.168.70.5 has m.2 512GB 950 Pro. I am using a 2.5" 512GB 850 Pro for cache on the vm image of 192.168.70.22. This 850 Pro is in the ESXi server.

So now tune the 10Gb image:

---turn off NetworkManage     

systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl stop NetworkManager-wait-online
systemctl disable NetworkManager-wait-online
systemctl enable network
chkconfig network on
systemctl start network


reboot and sanity check

systemctl status NetworkManager
systemctl status network

---end turn off NetworkManager

---tune 10Gb CNA

service irqbalance stop
service cpuspeed stop
chkconfig irqbalance off
chkconfig cpuspeed off
systemctl disable irqbalance
systemctl disable cpuspeed
systemctl stop irqbalance
systemctl stop cpuspeed

vi /etc/sysconfig/network-scripts/ifcfg-eth???
MTU="9000"

vi /etc/sysctl.conf
# -- tuning -- #
# Increase system file descriptor limit
fs.file-max = 65535

# Increase system IP port range to allow for more concurrent connections
net.ipv4.ip_local_port_range = 1024 65000

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

---end tune 10Gb CNA 

Thursday, February 23, 2017

Asterisk 14 on Centos 7.3

Asterisk 14 on Centos 7.3

After fresh of install of Centos 7.3 minimal.

cd /root

cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux

cp /etc/selinux/config /etc/selinux/config.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

systemctl disable firewalld
systemctl stop firewalld

service iptables stop
service ip6tables stop
chkconfig iptables off
chkconfig ip6tables off

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git vim

yum -y groupinstall "Development Tools"
yum -y update
yum -y upgrade

cd /root
echo ':color desert' > .vimrc


reboot

cd /root

yum -y install epel-release
yum -y install stress htop iftop iotop hddtemp smartmontools iperf3 sysstat mlocate
updatedb

yum -y install gcc gcc-c++ php-xml php php-mysql php-pear php-mbstring mariadb-devel mariadb-server mariadb sqlite-devel lynx bison gmime-devel psmisc tftp-server httpd make ncurses-devel libtermcap-devel sendmail sendmail-cf caching-nameserver sox newt-devel libxml2-devel libtiff-devel audiofile-devel gtk2-devel uuid-devel libtool libuuid-devel subversion kernel-devel kernel-devel-$(uname -r) git subversion kernel-devel php-process crontabs cronie cronie-anacron wget vim

wget http://www.digip.org/jansson/releases/jansson-2.7.tar.gz

tar -xzvf jansson-2.7.tar.gz

cd jansson-2.7

./configure -prefix=/usr

make clean

make && make install

ldconfig

cd /root

wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-14-current.tar.gz

tar xvf asterisk-14-current.tar.gz

cd asterisk-14.3.0

./configure -libdir=/usr/lib64

make menuselect

contrib/scripts/get_mp3_source.sh

make

make install

make progdocs

make samples

make config

systemctl restart asterisk

systemctl status asterisk

asterisk -r

Sunday, February 19, 2017

Broadsoft useful commands

Broadsoft useful commands

Backup servers:

bwadmin@ns1.whittenberg.domain$ locate bwBackup.pl

/usr/local/broadworks/NS_Rel_20.sp1_1.606/bin/bwBackup.pl

bwadmin@ns1.whittenberg.domain$ /usr/local/broadworks/NS_Rel_20.sp1_1.606/bin/bwBackup.pl NetworkServer /tmp/NetworkServer.bak

via the cli is:
NS_CLI/Maintenance/Tools> backupdb /tmp/NetworkServer.bak

Also see
bwRestore.pl
NS_CLI/Maintenance/Tools> restoredb

owncloud helpful commands

Add a trusted domain.:
http://192.168.80.100/owncloud/index.php/settings/admin?trustDomain=192.168.10.100

Clean trashbin/deleted files:
sudo -u apache /var/www/html/owncloud/occ trashbin:cleanup
sudo -u apache /var/www/html/owncloud/occ trashbin:cleanup user1 user2

Take out of maintenance mode:

vim /var/www/html/owncloud/config/config.php

<?php
$CONFIG = array (
  'updatechecker' => false,
  'instanceid' => 'oce24qfiihel',
  'passwordsalt' => 'Dy+BHYATWwB+F4MYYBjPjYX44iXB3o',
  'secret' => 'Bpf9UEgisK4XJecVxrL76ZAwg4n9NnG6RXWO2B4MKzpJNGMz',
  'trusted_domains' =>
  array (
    0 => '192.168.80.100',
    1 => '192.168.10.100',
  ),
  'datadirectory' => '/myraid/owncloud/',
  'overwrite.cli.url' => 'http://192.168.80.100/owncloud',
  'dbtype' => 'sqlite3',
  'version' => '9.1.4.2',
  'logtimezone' => 'UTC',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

Friday, February 17, 2017

owncloud take out of maintenance mode

owncloud take out of maintenance mode

vim /var/www/html/owncloud/config/config.php

<?php
$CONFIG = array (
  'updatechecker' => false,
  'instanceid' => 'oce24qfiihel',
  'passwordsalt' => 'Dy+BHYATWwB+F4MYYBjPjYX44iXB3o',
  'secret' => 'Bpf9UEgisK4XJecVxrL76ZAwg4n9NnG6RXWO2B4MKzpJNGMz',
  'trusted_domains' =>
  array (
    0 => '192.168.80.100',
    1 => '192.168.10.100',
  ),
  'datadirectory' => '/myraid/owncloud/',
  'overwrite.cli.url' => 'http://192.168.80.100/owncloud',
  'dbtype' => 'sqlite3',
  'version' => '9.1.4.2',
  'logtimezone' => 'UTC',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

Thursday, February 9, 2017

Install Linux Kernel 4.9 In CentOS

Install Linux Kernel 4.9 In CentOS

Ripped from: https://www.ostechnix.com/install-linux-kernel-4-9-centos-ubuntu/ 

 

To install the latest kernel, we need to add ELRepo repository.

First,  let us add ELRepo GPG key:
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Then, add ELRepo in CentOS 7 / RHEL 7 / Scientific Linux 7 using command:
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

To Install ELRepo for CentOS 6 / RHEL 6 / Scientific Linux 6:
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
To Install ELRepo for CentOS 5 / RHEL 5 / Scientific Linux 6:
rpm -Uvh http://www.elrepo.org/elrepo-release-5-5.el5.elrepo.noarch.rpm

Enable ELRepo fastest mirror by installing the following package:
yum install yum-plugin-fastestmirror

Sample output:
 
[...]
Dependencies Resolved

================================================================================
 Package Arch Version Repository
 Size
================================================================================
Updating:
 yum-plugin-fastestmirror noarch 1.1.31-40.el7 base 32 k

Transaction Summary
================================================================================
Upgrade 1 Package

Total download size: 32 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for base
yum-plugin-fastestmirror-1.1.31-40.el7.noarch.rpm | 32 kB 00:02 
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
 Updating : yum-plugin-fastestmirror-1.1.31-40.el7.noarch 1/2 
 Cleanup : yum-plugin-fastestmirror-1.1.31-34.el7.noarch 2/2 
 Verifying : yum-plugin-fastestmirror-1.1.31-40.el7.noarch 1/2 
 Verifying : yum-plugin-fastestmirror-1.1.31-34.el7.noarch 2/2 

Updated:
 yum-plugin-fastestmirror.noarch 0:1.1.31-40.el7 

Complete!

We have added the ELRepo. Now, it is time to install Linux kernel 4.9.

Let us find out the current version details with command:
 
uname -r

Sample output:
3.10.0-327.22.2.el7.x86_64

So, the currently installed Linux Kernel version is 3.10.0-327.22.2.el7.x86_64.

Now, let us install Linux Kernel 4.9 by running the following command from Terminal:
yum --enablerepo=elrepo-kernel install kernel-ml


After installing the Kernel, Reboot your system and select the latest Kernel from the Grub boot menu.


Once you successfully logged-in to the system, run the following command to verify whether latest the latest Linux Kernel has been installed or not:
 
uname -r

Sample output:
4.9.0-1.el7.elrepo.x86_64


Congratulations! You have successfully updated the Linux Kernel. Now, your system is powered by most recent stable Linux Kernel 4.9 version.

If have encountered with any problems after installing the new Linux Kernel? No worries, reboot your system. Select your previously working Kernel from the Boot menu.


Finally, remove the newly installed Kernel using command:
 
yum remove kernel-ml

That’s it.

Tuesday, January 17, 2017

ESXi 6 install stuck on “Relocating modules and starting up the kernel…




ESXi 6 install stuck on “Relocating modules and starting up the kernel…”


Admittedly, I do not do much VMWare these days, but on occasion I have clients that need to use it. Recently I was installing ESXi 6  on my personal lab to do some testing, when the installation got stuck, the last message being  “Relocating modules and starting up the kernel”

image

So, what’s happening? Granted this is an older Sun 4170M2 server, but  I have installed Oracle VM on the server with no issues. I also have run KVM hypervisors, Windows Hyper-V and Citrix without any issues. Well after a little troubleshooting it looks like VMWare does not like the video subsystem and is getting confused when it is checking to see if the server is headless.

The fix is relatively easy, we have to add a boot parameter to the ESXi kernel. To do this, reboot, and while ESXi is booting hit the tab key

image

Next, we will add a parameter to disable the headless check “ignoreHeadless=TRUE”

image

And now reboot and install normally…

image

Continue the installation normally, but when you get to the end pay attention.

image

Now reboot, and get ready to hit “Shift O” this time

image

The add in the ignoreHeadless=TRUE in the boot parameter list, it should look like this

image

And then hit enter to continue the boot process.
Next, once we are back up we will need to make the ignoreHeadless=TRUE permanent for each reboot.

Log into the ESXi system as root, from the login screen hit F2

image

Login as root

image

Arrow down to troubleshooting options to enable the shell

image

Next enable the ESXi shell

image

The enter key will toggle the shell state.

image

Now from the console hit ALT-F2. On ILOMs you will need to use enable the ALT key from the keyboard menu

image

Then hit F2, to get to the shell login.
Don’t forget to uncheck the alt key from the keyboard dropdown once you are on the shell screen, otherwise you will have issues typing in the password!

image

next login as root, and we will run a single command
esxcfg-advcfg --set-kernel "TRUE" ignoreHeadless

image

I am a big fan of the concept of “trust, but verify”, so let’s verify that the parameter is correct by getting the parameter with the command
“esxcfg-advcfg –get-kernel ignoreHeadless”

image

Now we  are done. type exit to logout, and then ALT-F2 to go back to the main screen

Hopefully this helps you with getting ESX installed on some older systems.



Tuesday, October 25, 2016

CentOS 7 sshd server to server with no password using trusted keys

CentOS 7 sshd server to server with no password using trusted keys

I have two servers of 192.168.10.28 and 192.168.10.100. On 192.168.10.28 as root:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a3:ba:98:53:bc:8c:c9:b3:d8:3f:73:e2:61:a6:41:f7 root@rhtest.whittenberg.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
|  ...   S        |
| . .o. . .       |
| ..=+.E          |
| o*B*o.          |
|. **=*           |
+-----------------+

# ssh-copy-id root@192.168.10.100
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.100's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.100'"
and check to make sure that only the key(s) you wanted were added.

# ssh root@192.168.10.100
Last login: Tue Oct 25 20:09:25 2016 from 192.168.10.28


Now do the same on 192.168.10.100 using the ssh-copy-id command to host 192.168.10.28 like the following:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a3:ba:98:53:bc:8c:c9:b3:d8:3f:73:e2:61:a6:41:f7 root@rhtest.whittenberg.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
|  ...   S        |
| . .o. . .       |
| ..=+.E          |
| o*B*o.          |
|. **=*           |
+-----------------+

# ssh-copy-id root@192.168.10.28
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.100's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.28'"
and check to make sure that only the key(s) you wanted were added.

# ssh root@192.168.10.28
Last login: Tue Oct 25 20:09:25 2016 from 192.168.10.100


There you go. You you no longer need password to use the sshd protocol between your two servers. Try this on either server access the other. I will be on the .28 accessing the .100:

# sftp root@192.168.10.100
Connected to 192.168.10.100.
sftp> ls
anaconda-ks.cfg                    btrfs-progs                        dead.letter                        dump
fio-2.0.9.tar.gz                   go.sh                              idle3-tools-0.9.1                  idle3-tools-0.9.1.tar
linux-4.7.tar.xz                   list                               master.zip                         mylog.txt
phoronix-test-suite                phoronix-test-suite-6.4.0.tar      scrub.log                          sipp-3.5.1
sipp-3.5.1.tar.gz                  test.sh                            trantest                           wd5741x64
zfsstress-master
sftp>

Yup. No password needed. Enjoy!


Saturday, October 22, 2016

CentOS 7 rescue, backup, and restore using ReaR

CentOS 7 rescue, backup, and restore using ReaR

I will be using a server i have already setup as my backup server. The client server we are working on in this post will use the backup server to store its rescue image and tar file on. This backup server is connected via 10Gb Ethernet and has 8 WD 3TB NAS drives in btrfs Raid10 on a Supermicro SAS/SATA controller. The btrfs on this backup server is also replicated to another server. So if a Linux client crashed i have what i need to restore from my backup server. If the back server crashes before i get the client rebuilt i can fall back on the replicated files the other machine. I also have a disaster recovery document for the backup server that allows me to have it fully rebuilt and able to restore files back on to it in less than an hour.

Now on your client machine install files so we can create rescue images. lftp is needed to so we can sftp the image to the backup server if wanted. In this post i will be using nfs as i have this setup already on my backup server.

# yum -y install rear genisoimage syslinux lftp nfs-utils

ReaR is configured in the /etc/rear/local.conf file. Specify the rescue system configuration by
adding these lines:

OUTPUT=output format
OUTPUT_URL=output location


Substitute output format with rescue system format, for example, ISO for an ISO disk image or USB for a bootable USB.

Substitute output location with where it will be put, for example,
file:///mnt/rescue_system/ for a local filesystem directory or
sftp://backupuser:password@192.168.10.100/directory/ for an sftp directory or.
nfs://192.168.10.100/backup_dir/ for a nfs share.

Example1:
OUTPUT=ISO
OUTPUT_URL=sftp://backup:password@192.168.10.100/backup_dir/


Example2:
OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/


In the exmaple2 above i create the rescue ISO image on the server 192.168.10.100 using my nfs share /backup_dir.

Make rescure image with:

# rear -v mkrescue

Now look under /backup_dir on server 192.168.10.100 and you will see the newly created directory of your machine name with the ISO image in it.

If you like you can verify that you can boot from the ISO. Burn it to a CD/DVD, or like me, just create a new vm image and have this as the ISO CD/DVD and set to boot on start.

I can now boot off the rescue ISO but this does not do me much good if lets say i had a drive crash and all my files are gone. So say i have replaced the drive and I will also need to restore them to have a functional system again.

So lets add the option in the /etc/rear/local.conf file to tar a backup file. Our /etc/rear/local.conf file already contains the following:

OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/


Now lets append:

BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_URL=nfs://192.168.10.100/backup_dir/
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/mnt/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"


So the complete file looks like:

OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/
BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_URL=nfs://192.168.10.100/backup_dir/
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/mnt/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"


So i am using tar for my backup program, my backups will stored via nfs share to my backup server under the /backup_dir/, i will gzip the tar with extension .gz, and exclude /tmp/*, /dev/shm/*, and /mnt/* from the backups.

Lets test. Now run:

# rear -v mkbackup

After completed you should see the not only the rescue Image under /backup_dir/ on the backup server, but now the backup.tar.gz file as well.

Now to go through the rescue and restore procedure.

In this example i will use my vSphere ESXi 60 server and create a new image for this. I have uploaded the rescue ISO to a datastore and have the image mount this ISO at boot, so i can boot from it.

After the vm image is booted i am presented with the ReaR menu.

I choose Recover and press enter.
Login as root (no password required).
At the # prompt type in:

# rear recover
 
If asked if you want to restore your backup say y.

Now reboot server and login.

Enjoy!

Friday, October 21, 2016

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

This is in preparation of me studying for my RHCSA (Redhat Certified System Administrator)

------------------------------------
Now for the base server installs.
------------------------------------

I will using the CentOS-7-x86_64-Everything.iso for the installs in my VMware vSphere Client 6.0 and ESXi 6.0. Each Server gets 2 Cores, 2GB Ram, 25GB disk, and 1 NIC.

Servers names and IP addresses will be:
rh-ipa.whittenberg.local 192.168.10.30
rh-srv1.whittenberg.local 192.168.10.31
rh-srv2.whittenberg.local 192.168.10.32

My Local LAN router is 192.168.10.1 and i will use this for my gateway out and DNS server.

At the CentOS 7 install I set my Network and Hostname.

Click on Network and Hostname on home screen.
Hostname: rh-ipa.whittenberg.local
Click slider to turn ethernet adapter ON. This gives me an IP from my DHCP server from my local LAN router.
Click the Configure button.
[General]
click on Automatically connect to this network
[IPv4 Settings]
Method: Manual
Addresses, click add button
address: 192.168.10.30
netmask: 24

gateway: 192.168.10.1
DNS server: 192.168.10.1
Click Save button
Click Done button

You should now be able to ping 192.168.10.30 from your workstation on the LAN.

Click on  Date & Time from the home screen.
Chose your region and city.
Make sure Network Time slider shows ON.
Click on gears next to slider. Time servers should be green. If red then you are unable to reach them.
Click Done button.

Click on Software Selection from home screen.
Select Minimal Install under Base Environment and nothing under Add-Ons.
Click Done button.

Click on Installation Destination from home screen.
Make sure white check mark on your hard disk.
Select I will configure partitioning.
Click Done button.
Click the + button and setup as follows.
/boot = 500MB
swap = 1024MB
/root = 23.5GB
Click Done button.
Accept Changes.

Now click the Begin Installation button.

Supply root password while the installation is taking place.

When finished click the Reboot button.

Putty/SSH into your server from your workstation.

Disable selinux.

cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux
cat /etc/sysconfig/selinux


reboot

Disable and turn off firewalld.
  
systemctl disable firewalld
systemctl stop firewalld


cat or edit your /etc/hostname file and add the following entree if not there already.

ipa.whittenberg.local

cat or edit your /etc/hosts file and add the following entries if not there already.

192.168.10.30    ipa.whittenberg.local ipa

cat or edit your /etc/resolv.conf file and add the following entries if not there already.

search whittenberg.local
nameserver 192.168.10.1


Install additional packages I need on normal basis.

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git

yum -y groupinstall "Development Tools"


Now update the server.

yum -y update

reboot

**Optional - Install of GNOME Desktop.
yum -y groups install "GNOME Desktop"
yum -y update
systemctl set-default graphical.target
systemctl get-default


reboot and make sure GNOME Desktop starts.
**END Optional - Install of GNOME Desktop.

**Optional - Turn off console screen saver and display all boot messages.
vi /etc/default/grub and on the line starting with GRUB_CMDLINE_LINUX= remove 'rhgb' and 'quiet'. Now add 'consoleblank=0' just before the last " on the line. DO NOT include the ' in the 'consoleblank=0' add.

Now make a new grub.cfg using the line below.

grub2-mkconfig -o /boot/grub2/grub.cfg

If using efi then use this line to make new grub.cfg

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg


Now when you boot your server you will see all the boot messages.
**END Optional - Turn off console screen saver and display all boot messages.


NOW REPEAT all the steps above under base server installs for the other two server of rh-srv1 and rh-srv2, of course using the correct IP and hostname for each server.

------------------------------------
Now for the ipa server install
------------------------------------

Now its time to setup ipa on the ipa server and we will user srv1 as our first client.

On your ipa server.

# yum -y install ipa-server bind-dyndb-ldap ipa-server-dns

# ipa-server-install --setup-dns


The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
  * Configure a stand-alone CA (dogtag) for certificate management
  * Configure the Network Time Daemon (ntpd)
  * Create and configure an instance of Directory Server
  * Create and configure a Kerberos Key Distribution Center (KDC)
  * Configure Apache (httpd)
  * Configure DNS (bind)

To accept the default shown in brackets, press the Enter key.

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
<hostname>.<domainname>
Example: master.example.com.


Server host name [rh-ipa.whittenberg.local]:

Warning: skipping DNS resolution of host rh-ipa.whittenberg.local
The domain name has been determined based on the host name.

Please confirm the domain name [whittenberg.local]:

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [WHITTENBERG.LOCAL]:
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password:
Password (confirm):

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password:
Password (confirm):

Existing BIND configuration detected, overwrite? [no]: yes
Do you want to configure DNS forwarders? [yes]:
Enter an IP address for a DNS forwarder, or press Enter to skip: 8.8.8.8
DNS forwarder 8.8.8.8 added. You may add another.
Enter an IP address for a DNS forwarder, or press Enter to skip:
Checking DNS forwarders, please wait ...
Do you want to configure the reverse zone? [yes]:
Please specify the reverse zone name [10.168.192.in-addr.arpa.]:
Using reverse zone(s) 10.168.192.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:       rh-ipa.whittenberg.local
IP address(es): 192.168.10.30
Domain name:    whittenberg.local
Realm name:     WHITTENBERG.LOCAL

BIND DNS server will be configured to serve IPA domain with:
Forwarders:    8.8.8.8
Reverse zone(s):  10.168.192.in-addr.arpa.

Continue to configure the system with these values? [no]: yes

Install info scrolls by then at the end we see.

Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind
                  * 123: ntp

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

# kinit admin
Password for admin@WHITTENBERG.LOCAL:

# ipa user-find admin

--------------
1 user matched
--------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  UID: 1546000000
  GID: 1546000000
  Account disabled: False
  Password: True
  Kerberos keys available: True
----------------------------
Number of entries returned 1
----------------------------

# ipa user-add
First name: ldapuser1
Last name: ldapuser1
User login [lldapuser1]: ldapuser1
----------------------
Added user "ldapuser1"
----------------------
  User login: ldapuser1
  First name: ldapuser1
  Last name: ldapuser1
  Full name: ldapuser1 ldapuser1
  Display name: ldapuser1 ldapuser1
  Initials: ll
  Home directory: /home/ldapuser1
  GECOS: ldapuser1 ldapuser1
  Login shell: /bin/sh
  Kerberos principal: ldapuser1@WHITTENBERG.LOCAL
  Email address: ldapuser1@whittenberg.local
  UID: 1546000001
  GID: 1546000001
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False
 
# ls -lsa /etc/ipa/
total 24
 0 drwxr-xr-x.   6 root root   89 Oct 20 20:02 .
12 drwxr-xr-x. 147 root root 8192 Oct 20 20:02 ..
 4 -r--r--r--.   1 root root 1337 Oct 20 20:01 ca.crt
 4 -rw-r--r--.   1 root root  321 Oct 20 20:00 default.conf
 0 drwxr-xr-x.   2 root root   76 Oct 20 20:02 dnssec
 4 drwxr-xr-x.   2 root root 4096 Oct 20 19:52 html
 0 drwxr-xr-x.   2 root root   51 Oct 20 20:01 kdcproxy
 0 drwxr-xr-x.   2 root root   69 Oct 20 20:02 nssdb

# cat /etc/ipa/ca.crt
-----BEGIN CERTIFICATE-----
MIIDoDCCAoigAwIBAgIBATANBgkqhkiG9w0BAQsFADA8MRowGAYDVQQKDBFXSElU
VEVOQkVSRy5MT0NBTDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4X
DTE2MTAyMTAzMDAwNFoXDTM2MTAyMTAzMDAwNFowPDEaMBgGA1UECgwRV0hJVFRF
TkJFUkcuTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANC2wYqkiEwRY/YUY5cyDTEh+OaK
tU8KtPEeyAOhE8COugNYUZhx8bvoJyM7c7KGzxU/nuY4MAYF6sMOslIcaAHR7S4A
Wi+1URBFI8xsCoO7YktVoyTkV0sNY93dUlK+dpKNnL7P1oDJlcqDWflne/BG0h4i
gq5NRRdWQvh33yJAriBNSw2AIYvuQMiMQ1kdFqQ/T5VjQUU+L20QGE2Ox7AnHkYd
Tovn+Alok8FIqbDNqIn4VNsz0Zy010eHC7zmRK/kiWrnauf+r/JHi3zGz9GVKtUE
3swEE4dXejA7SAd03ZnmF2oruJ+o5u3ZwK4bN8S9dKuZZTaXxjSbT+FbM3MCAwEA
AaOBrDCBqTAfBgNVHSMEGDAWgBRO+0W/EXt3Qbkun00PARxow2PUvjAPBgNVHRMB
Af8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUTvtFvxF7d0G5Lp9N
DwEcaMNj1L4wRgYIKwYBBQUHAQEEOjA4MDYGCCsGAQUFBzABhipodHRwOi8vcmgt
aXBhLndoaXR0ZW5iZXJnLmxvY2FsOjgwL2NhL29jc3AwDQYJKoZIhvcNAQELBQAD
ggEBAGGiKMrGrcCAtRVfKdA2f4Ko1FpIKKUsEMG4yyBxio//HxDja8X0MUzG7RWQ
3xKvkcNO08zFNd5/5YCiboy5RRrz4N/fIVFZDjJhLjwT9UQx1nktETLl/oirVSp4
e3z7Tc6m+4gpPABM7ryVNkQfdNCka82rLnzny5+U/YsZ2mGcl86MQ6T2U/0RXBHj
9QZVmVRJe7mTJzNW62rSsh6D7u2RBffZIQ7I/Huk5HFiDI9mogtizpb1Miwr5Vhb
uy+bVmPaHJfy/bHbVv8Vg3m0dKbP6bhS0lafnfWXF1iCmako0n0v2+T7IzwScW2Y
G94EkYeftqGg9LiFAogEEKZnuGg=
-----END CERTIFICATE-----

# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@WHITTENBERG.LOCAL

Valid starting       Expires              Service principal
10/20/2016 20:09:13  10/21/2016 20:07:42  HTTP/rh-ipa.whittenberg.local@WHITTENBERG.LOCAL
10/20/2016 20:07:48  10/21/2016 20:07:42  krbtgt/WHITTENBERG.LOCAL@WHITTENBERG.LOCAL

# cat /etc/resolv.conf
search whittenberg.local
nameserver 127.0.0.1


vi /etc/resolv.conf (change to nameserver 192.168.10.30)

# cat /etc/resolv.conf
search whittenberg.local
nameserver 192.168.10.30


Next you can use the Firefox browser on your ipa server and browse the server.

https://rh-ipa.whittenberg.local

Supply username and password. User name is admin.

Click on Network Services then DNS->DNS Zones, then click on whittenberg.local.

Click Add button. Add your servers here like rh-srv1 and rh-srv2. include name, IP address, and select Create Reverse. Like:

name = rh-srv1
Address = 192.168.10.31
Check mark on Create Reverse

and

name = rh-srv2
Address = 192.168.10.32
Check mark on Create Reverse

------------------------------------
Now for the ipa client installs
------------------------------------

On rh-srv1.whittenberg.local server.

[root@rh-srv1 ~]# cat /etc/hostname
rh-srv1.whittenberg.local

[root@rh-srv1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.31   rh-srv1.whittenberg.local rh-srv1

[root@rh-srv1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search whittenberg.local
nameserver 192.168.10.1
nameserver 2001:578:3f::30
nameserver 2001:578:3f:1::30

Now change the nameserver in /etc/resolv.conf from 192.168.10.1, as shown above, to 192.168.10.30, as shown below.

[root@rh-srv1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search whittenberg.local
nameserver 192.168.10.30
nameserver 2001:578:3f::30
nameserver 2001:578:3f:1::30


Install the ipa client.

[root@rh-srv1 ~]# yum -y install ipa-client

Now run the install/setup of the ipa client.

[root@rh-srv1 ~]# ipa-client-install
WARNING: ntpd time&date synchronization service will not be configured as
conflicting service (chronyd) is enabled
Use --force-ntpd option to disable it and force configuration of ntpd

Discovery was successful!
Client hostname: rh-srv1.whittenberg.local
Realm: WHITTENBERG.LOCAL
DNS Domain: whittenberg.local
IPA Server: rh-ipa.whittenberg.local
BaseDN: dc=whittenberg,dc=local

Continue to configure the system with these values? [no]: yes
Skipping synchronizing time with NTP server.
User authorized to enroll computers: admin
Password for admin@WHITTENBERG.LOCAL:
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=WHITTENBERG.LOCAL
    Issuer:      CN=Certificate Authority,O=WHITTENBERG.LOCAL
    Valid From:  Fri Oct 21 03:00:04 2016 UTC
    Valid Until: Tue Oct 21 03:00:04 2036 UTC

Enrolled in IPA realm WHITTENBERG.LOCAL
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm WHITTENBERG.LOCAL
trying https://rh-ipa.whittenberg.local/ipa/json
Forwarding 'ping' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
Systemwide CA database updated.
Added CA certificates to the default NSS database.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring whittenberg.local as NIS domain.
Client configuration complete.



Setup the server to auto create home directories for ipa users when first login from the server.

[root@rh-srv1 ~]# authconfig --enablemkhomedir --update
getsebool:  SELinux is disabled



Now logout as local user root and back in as ipa user admin.

[root@rh-srv1 ~]# logout

login as: admin
admin@192.168.10.31's password:
Creating home directory for admin.
[admin@rh-srv1 ~]$

[admin@rh-srv1 ~]$ pwd
/home/admin

[admin@rh-srv1 ~]$ whoami
admin

[admin@rh-srv1 ~]$ ping rh-ipa
PING rh-ipa.whittenberg.local (192.168.10.30) 56(84) bytes of data.
64 bytes from rh-ipa.whittenberg.local (192.168.10.30): icmp_seq=1 ttl=64 time=0.084 ms
64 bytes from rh-ipa.whittenberg.local (192.168.10.30): icmp_seq=2 ttl=64 time=0.084 ms
^C
--- rh-ipa.whittenberg.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.084/0.084/0.084/0.000 ms

Now you should be able to ssh into the rh-ipa server without supplying credentials if still login as admin on rh-srv1.

[admin@rh-srv1 ~]$ ssh rh-ipa
Could not chdir to home directory /home/admin: No such file or directory
-bash-4.2$


It worked! Looks like we have home dir issue for ipa user admin on the ipa server though. Lets try and take care of that. First exit back to rh-srv1.

-bash-4.2$ exit
logout
Connection to rh-ipa closed.
[admin@rh-srv1 ~]$


Now ssh to rh-ipa as local user root.

[admin@rh-srv1 ~]$ ssh root@rh-ipa
root@rh-ipa's password:
Last failed login: Fri Oct 21 22:40:19 MST 2016 from 192.168.10.31 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Thu Oct 20 21:19:20 2016 from 192.168.10.5
[root@rh-ipa ~]#


Setup the server to auto create home directories for ipa users when first login from the server.

[root@rh-srv1 ~]# authconfig --enablemkhomedir --update
Now exit back to rh-srv1.

[root@rh-ipa ~]# exit
logout
Connection to rh-ipa closed.
[admin@rh-srv1 ~]$


Now ssh back in to rh-ipa as ipa user admin.

[admin@rh-srv1 ~]$ ssh rh-ipa
Creating home directory for admin.
Last login: Fri Oct 21 22:36:53 2016 from 192.168.10.31
[admin@rh-ipa ~]$ pwd
/home/admin
[admin@rh-ipa ~]$



That solved the problem!!


NOW REPEAT all the steps above in ipa client install section for rh-srv2, of course using the correct IP and hostname for each server.

Thats it folks. Enjoy!