Broadsoft, Adtran, and Centos: 2016

Tuesday, October 25, 2016

CentOS 7 sshd server to server with no password using trusted keys

CentOS 7 sshd server to server with no password using trusted keys

I have two servers of 192.168.10.28 and 192.168.10.100. On 192.168.10.28 as root:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a3:ba:98:53:bc:8c:c9:b3:d8:3f:73:e2:61:a6:41:f7 root@rhtest.whittenberg.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
| ...   S        |
| . .o. . .       |
| ..=+.E          |
| o*B*o.          |
|. **=*           |
+-----------------+

# ssh-copy-id root@192.168.10.100
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.100's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.100'"
and check to make sure that only the key(s) you wanted were added.

# ssh root@192.168.10.100
Last login: Tue Oct 25 20:09:25 2016 from 192.168.10.28

Now do the same on 192.168.10.100 using the ssh-copy-id command to host 192.168.10.28 like the following:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
a3:ba:98:53:bc:8c:c9:b3:d8:3f:73:e2:61:a6:41:f7 root@rhtest.whittenberg.local
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|                 |
| ...   S        |
| . .o. . .       |
| ..=+.E          |
| o*B*o.          |
|. **=*           |
+-----------------+

# ssh-copy-id root@192.168.10.28
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.10.100's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@192.168.10.28'"
and check to make sure that only the key(s) you wanted were added.

# ssh root@192.168.10.28
Last login: Tue Oct 25 20:09:25 2016 from 192.168.10.100

There you go. You you no longer need password to use the sshd protocol between your two servers. Try this on either server access the other. I will be on the .28 accessing the .100:

# sftp root@192.168.10.100
Connected to 192.168.10.100.
sftp> ls
anaconda-ks.cfg                    btrfs-progs                        dead.letter                        dump
fio-2.0.9.tar.gz                   go.sh                              idle3-tools-0.9.1                  idle3-tools-0.9.1.tar
linux-4.7.tar.xz                   list                               master.zip                         mylog.txt
phoronix-test-suite                phoronix-test-suite-6.4.0.tar      scrub.log                          sipp-3.5.1
sipp-3.5.1.tar.gz                  test.sh                            trantest                           wd5741x64
zfsstress-master
sftp>
Yup. No password needed. Enjoy!

Saturday, October 22, 2016

CentOS 7 rescue, backup, and restore using ReaR

CentOS 7 rescue, backup, and restore using ReaR

I will be using a server i have already setup as my backup server. The client server we are working on in this post will use the backup server to store its rescue image and tar file on. This backup server is connected via 10Gb Ethernet and has 8 WD 3TB NAS drives in btrfs Raid10 on a Supermicro SAS/SATA controller. The btrfs on this backup server is also replicated to another server. So if a Linux client crashed i have what i need to restore from my backup server. If the back server crashes before i get the client rebuilt i can fall back on the replicated files the other machine. I also have a disaster recovery document for the backup server that allows me to have it fully rebuilt and able to restore files back on to it in less than an hour.

Now on your client machine install files so we can create rescue images. lftp is needed to so we can sftp the image to the backup server if wanted. In this post i will be using nfs as i have this setup already on my backup server.

# yum -y install rear genisoimage syslinux lftp nfs-utils

ReaR is configured in the /etc/rear/local.conf file. Specify the rescue system configuration by
adding these lines:

OUTPUT=output format
OUTPUT_URL=output location

Substitute output format with rescue system format, for example, ISO for an ISO disk image or USB for a bootable USB.

Substitute output location with where it will be put, for example,
file:///mnt/rescue_system/ for a local filesystem directory or
sftp://backupuser:password@192.168.10.100/directory/ for an sftp directory or.
nfs://192.168.10.100/backup_dir/ for a nfs share.

Example1:
OUTPUT=ISO
OUTPUT_URL=sftp://backup:password@192.168.10.100/backup_dir/

Example2:
OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/

In the exmaple2 above i create the rescue ISO image on the server 192.168.10.100 using my nfs share /backup_dir.

Make rescure image with:

# rear -v mkrescue

Now look under /backup_dir on server 192.168.10.100 and you will see the newly created directory of your machine name with the ISO image in it.

If you like you can verify that you can boot from the ISO. Burn it to a CD/DVD, or like me, just create a new vm image and have this as the ISO CD/DVD and set to boot on start.

I can now boot off the rescue ISO but this does not do me much good if lets say i had a drive crash and all my files are gone. So say i have replaced the drive and I will also need to restore them to have a functional system again.

So lets add the option in the /etc/rear/local.conf file to tar a backup file. Our /etc/rear/local.conf file already contains the following:

OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/

Now lets append:

BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_URL=nfs://192.168.10.100/backup_dir/
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/mnt/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"

So the complete file looks like:

OUTPUT=ISO
OUTPUT_URL=nfs://192.168.10.100/backup_dir/
BACKUP=NETFS
BACKUP_PROG=tar
BACKUP_URL=nfs://192.168.10.100/backup_dir/
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
BACKUP_PROG_COMPRESS_SUFFIX=".gz"
BACKUP_PROG_EXCLUDE=( '/tmp/*' '/dev/shm/*' '/mnt/*' )
BACKUP_OPTIONS="nfsvers=3,nolock"

So i am using tar for my backup program, my backups will stored via nfs share to my backup server under the /backup_dir/, i will gzip the tar with extension .gz, and exclude /tmp/*, /dev/shm/*, and /mnt/* from the backups.

Lets test. Now run:

# rear -v mkbackup
After completed you should see the not only the rescue Image under /backup_dir/ on the backup server, but now the backup.tar.gz file as well.

Now to go through the rescue and restore procedure.

In this example i will use my vSphere ESXi 60 server and create a new image for this. I have uploaded the rescue ISO to a datastore and have the image mount this ISO at boot, so i can boot from it.

After the vm image is booted i am presented with the ReaR menu.

I choose Recover and press enter.
Login as root (no password required).
At the # prompt type in:

# rear recover

If asked if you want to restore your backup say y.

Now reboot server and login.

Enjoy!

Friday, October 21, 2016

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

This is in preparation of me studying for my RHCSA (Redhat Certified System Administrator)

------------------------------------
Now for the base server installs.
------------------------------------

I will using the CentOS-7-x86_64-Everything.iso for the installs in my VMware vSphere Client 6.0 and ESXi 6.0. Each Server gets 2 Cores, 2GB Ram, 25GB disk, and 1 NIC.

Servers names and IP addresses will be:
rh-ipa.whittenberg.local 192.168.10.30
rh-srv1.whittenberg.local 192.168.10.31
rh-srv2.whittenberg.local 192.168.10.32

My Local LAN router is 192.168.10.1 and i will use this for my gateway out and DNS server.

At the CentOS 7 install I set my Network and Hostname.

Click on Network and Hostname on home screen.
Hostname: rh-ipa.whittenberg.local
Click slider to turn ethernet adapter ON. This gives me an IP from my DHCP server from my local LAN router.
Click the Configure button.
[General]
click on Automatically connect to this network
[IPv4 Settings]
Method: Manual
Addresses, click add button
address: 192.168.10.30
netmask: 24

gateway: 192.168.10.1
DNS server: 192.168.10.1
Click Save button
Click Done button

You should now be able to ping 192.168.10.30 from your workstation on the LAN.

Click on Date & Time from the home screen.
Chose your region and city.
Make sure Network Time slider shows ON.
Click on gears next to slider. Time servers should be green. If red then you are unable to reach them.
Click Done button.

Click on Software Selection from home screen.
Select Minimal Install under Base Environment and nothing under Add-Ons.
Click Done button.

Click on Installation Destination from home screen.
Make sure white check mark on your hard disk.
Select I will configure partitioning.
Click Done button.
Click the + button and setup as follows.
/boot = 500MB
swap = 1024MB
/root = 23.5GB
Click Done button.
Accept Changes.

Now click the Begin Installation button.

Supply root password while the installation is taking place.

When finished click the Reboot button.

Putty/SSH into your server from your workstation.

Disable selinux.

cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux
cat /etc/sysconfig/selinux

reboot

Disable and turn off firewalld.

systemctl disable firewalld
systemctl stop firewalld

cat or edit your /etc/hostname file and add the following entree if not there already.

ipa.whittenberg.local

cat or edit your /etc/hosts file and add the following entries if not there already.

192.168.10.30    ipa.whittenberg.local ipa

cat or edit your /etc/resolv.conf file and add the following entries if not there already.

search whittenberg.local
nameserver 192.168.10.1

Install additional packages I need on normal basis.

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git

yum -y groupinstall "Development Tools"

Now update the server.

yum -y update

reboot

**Optional - Install of GNOME Desktop.
yum -y groups install "GNOME Desktop"
yum -y update
systemctl set-default graphical.target
systemctl get-default

reboot and make sure GNOME Desktop starts.
**END Optional - Install of GNOME Desktop.

**Optional - Turn off console screen saver and display all boot messages.
vi /etc/default/grub and on the line starting with GRUB_CMDLINE_LINUX= remove 'rhgb' and 'quiet'. Now add 'consoleblank=0' just before the last " on the line. DO NOT include the ' in the 'consoleblank=0' add.

Now make a new grub.cfg using the line below.

grub2-mkconfig -o /boot/grub2/grub.cfg

If using efi then use this line to make new grub.cfg

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Now when you boot your server you will see all the boot messages.
**END Optional - Turn off console screen saver and display all boot messages.

NOW REPEAT all the steps above under base server installs for the other two server of rh-srv1 and rh-srv2, of course using the correct IP and hostname for each server.

------------------------------------
Now for the ipa server install
------------------------------------

Now its time to setup ipa on the ipa server and we will user srv1 as our first client.

On your ipa server.

# yum -y install ipa-server bind-dyndb-ldap ipa-server-dns

# ipa-server-install --setup-dns

The log file for this installation can be found in /var/log/ipaserver-install.log
==============================================================================
This program will set up the IPA Server.

This includes:
* Configure a stand-alone CA (dogtag) for certificate management
* Configure the Network Time Daemon (ntpd)
* Create and configure an instance of Directory Server
* Create and configure a Kerberos Key Distribution Center (KDC)
* Configure Apache (httpd)
* Configure DNS (bind)

To accept the default shown in brackets, press the Enter key.

WARNING: conflicting time&date synchronization service 'chronyd' will be disabled
in favor of ntpd

Enter the fully qualified domain name of the computer
on which you're setting up server software. Using the form
<hostname>.<domainname>
Example: master.example.com.

Server host name [rh-ipa.whittenberg.local]:

Warning: skipping DNS resolution of host rh-ipa.whittenberg.local
The domain name has been determined based on the host name.

Please confirm the domain name [whittenberg.local]:

The kerberos protocol requires a Realm name to be defined.
This is typically the domain name converted to uppercase.

Please provide a realm name [WHITTENBERG.LOCAL]:
Certain directory server operations require an administrative user.
This user is referred to as the Directory Manager and has full access
to the Directory for system management tasks and will be added to the
instance of directory server created for IPA.
The password must be at least 8 characters long.

Directory Manager password:
Password (confirm):

The IPA server requires an administrative user, named 'admin'.
This user is a regular system account used for IPA server administration.

IPA admin password:
Password (confirm):

Existing BIND configuration detected, overwrite? [no]: yes
Do you want to configure DNS forwarders? [yes]:
Enter an IP address for a DNS forwarder, or press Enter to skip: 8.8.8.8
DNS forwarder 8.8.8.8 added. You may add another.
Enter an IP address for a DNS forwarder, or press Enter to skip:
Checking DNS forwarders, please wait ...
Do you want to configure the reverse zone? [yes]:
Please specify the reverse zone name [10.168.192.in-addr.arpa.]:
Using reverse zone(s) 10.168.192.in-addr.arpa.

The IPA Master Server will be configured with:
Hostname:       rh-ipa.whittenberg.local
IP address(es): 192.168.10.30
Domain name:    whittenberg.local
Realm name:     WHITTENBERG.LOCAL

BIND DNS server will be configured to serve IPA domain with:
Forwarders:    8.8.8.8
Reverse zone(s): 10.168.192.in-addr.arpa.

Continue to configure the system with these values? [no]: yes

Install info scrolls by then at the end we see.

Setup complete

Next steps:
        1. You must make sure these network ports are open:
                TCP Ports:
                  * 80, 443: HTTP/HTTPS
                  * 389, 636: LDAP/LDAPS
                  * 88, 464: kerberos
                  * 53: bind
                UDP Ports:
                  * 88, 464: kerberos
                  * 53: bind
                  * 123: ntp

        2. You can now obtain a kerberos ticket using the command: 'kinit admin'
           This ticket will allow you to use the IPA tools (e.g., ipa user-add)
           and the web user interface.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password

# kinit admin
Password for admin@WHITTENBERG.LOCAL:

# ipa user-find admin
--------------
1 user matched
--------------
User login: admin
Last name: Administrator
Home directory: /home/admin
Login shell: /bin/bash
UID: 1546000000
GID: 1546000000
Account disabled: False
Password: True
Kerberos keys available: True
----------------------------
Number of entries returned 1
----------------------------

# ipa user-add
First name: ldapuser1
Last name: ldapuser1
User login [lldapuser1]: ldapuser1
----------------------
Added user "ldapuser1"
----------------------
User login: ldapuser1
First name: ldapuser1
Last name: ldapuser1
Full name: ldapuser1 ldapuser1
Display name: ldapuser1 ldapuser1
Initials: ll
Home directory: /home/ldapuser1
GECOS: ldapuser1 ldapuser1
Login shell: /bin/sh
Kerberos principal: ldapuser1@WHITTENBERG.LOCAL
Email address: ldapuser1@whittenberg.local
UID: 1546000001
GID: 1546000001
Password: False
Member of groups: ipausers
Kerberos keys available: False

# ls -lsa /etc/ipa/
total 24
0 drwxr-xr-x.   6 root root   89 Oct 20 20:02 .
12 drwxr-xr-x. 147 root root 8192 Oct 20 20:02 ..
4 -r--r--r--.   1 root root 1337 Oct 20 20:01 ca.crt
4 -rw-r--r--.   1 root root 321 Oct 20 20:00 default.conf
0 drwxr-xr-x.   2 root root   76 Oct 20 20:02 dnssec
4 drwxr-xr-x.   2 root root 4096 Oct 20 19:52 html
0 drwxr-xr-x.   2 root root   51 Oct 20 20:01 kdcproxy
0 drwxr-xr-x.   2 root root   69 Oct 20 20:02 nssdb

# cat /etc/ipa/ca.crt
-----BEGIN CERTIFICATE-----
MIIDoDCCAoigAwIBAgIBATANBgkqhkiG9w0BAQsFADA8MRowGAYDVQQKDBFXSElU
VEVOQkVSRy5MT0NBTDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4X
DTE2MTAyMTAzMDAwNFoXDTM2MTAyMTAzMDAwNFowPDEaMBgGA1UECgwRV0hJVFRF
TkJFUkcuTE9DQUwxHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANC2wYqkiEwRY/YUY5cyDTEh+OaK
tU8KtPEeyAOhE8COugNYUZhx8bvoJyM7c7KGzxU/nuY4MAYF6sMOslIcaAHR7S4A
Wi+1URBFI8xsCoO7YktVoyTkV0sNY93dUlK+dpKNnL7P1oDJlcqDWflne/BG0h4i
gq5NRRdWQvh33yJAriBNSw2AIYvuQMiMQ1kdFqQ/T5VjQUU+L20QGE2Ox7AnHkYd
Tovn+Alok8FIqbDNqIn4VNsz0Zy010eHC7zmRK/kiWrnauf+r/JHi3zGz9GVKtUE
3swEE4dXejA7SAd03ZnmF2oruJ+o5u3ZwK4bN8S9dKuZZTaXxjSbT+FbM3MCAwEA
AaOBrDCBqTAfBgNVHSMEGDAWgBRO+0W/EXt3Qbkun00PARxow2PUvjAPBgNVHRMB
Af8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjAdBgNVHQ4EFgQUTvtFvxF7d0G5Lp9N
DwEcaMNj1L4wRgYIKwYBBQUHAQEEOjA4MDYGCCsGAQUFBzABhipodHRwOi8vcmgt
aXBhLndoaXR0ZW5iZXJnLmxvY2FsOjgwL2NhL29jc3AwDQYJKoZIhvcNAQELBQAD
ggEBAGGiKMrGrcCAtRVfKdA2f4Ko1FpIKKUsEMG4yyBxio//HxDja8X0MUzG7RWQ
3xKvkcNO08zFNd5/5YCiboy5RRrz4N/fIVFZDjJhLjwT9UQx1nktETLl/oirVSp4
e3z7Tc6m+4gpPABM7ryVNkQfdNCka82rLnzny5+U/YsZ2mGcl86MQ6T2U/0RXBHj
9QZVmVRJe7mTJzNW62rSsh6D7u2RBffZIQ7I/Huk5HFiDI9mogtizpb1Miwr5Vhb
uy+bVmPaHJfy/bHbVv8Vg3m0dKbP6bhS0lafnfWXF1iCmako0n0v2+T7IzwScW2Y
G94EkYeftqGg9LiFAogEEKZnuGg=
-----END CERTIFICATE-----

# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@WHITTENBERG.LOCAL

Valid starting       Expires              Service principal
10/20/2016 20:09:13 10/21/2016 20:07:42 HTTP/rh-ipa.whittenberg.local@WHITTENBERG.LOCAL
10/20/2016 20:07:48 10/21/2016 20:07:42 krbtgt/WHITTENBERG.LOCAL@WHITTENBERG.LOCAL

# cat /etc/resolv.conf
search whittenberg.local
nameserver 127.0.0.1

vi /etc/resolv.conf (change to nameserver 192.168.10.30)

# cat /etc/resolv.conf
search whittenberg.local
nameserver 192.168.10.30

Next you can use the Firefox browser on your ipa server and browse the server.

https://rh-ipa.whittenberg.local

Supply username and password. User name is admin.

Click on Network Services then DNS->DNS Zones, then click on whittenberg.local.

Click Add button. Add your servers here like rh-srv1 and rh-srv2. include name, IP address, and select Create Reverse. Like:

name = rh-srv1
Address = 192.168.10.31
Check mark on Create Reverse

and

name = rh-srv2
Address = 192.168.10.32
Check mark on Create Reverse

------------------------------------
Now for the ipa client installs
------------------------------------

On rh-srv1.whittenberg.local server.

[root@rh-srv1 ~]# cat /etc/hostname
rh-srv1.whittenberg.local

[root@rh-srv1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.10.31   rh-srv1.whittenberg.local rh-srv1

[root@rh-srv1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search whittenberg.local
nameserver 192.168.10.1
nameserver 2001:578:3f::30
nameserver 2001:578:3f:1::30
Now change the nameserver in /etc/resolv.conf from 192.168.10.1, as shown above, to 192.168.10.30, as shown below.

[root@rh-srv1 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search whittenberg.local
nameserver 192.168.10.30
nameserver 2001:578:3f::30
nameserver 2001:578:3f:1::30

Install the ipa client.

[root@rh-srv1 ~]# yum -y install ipa-client

Now run the install/setup of the ipa client.

[root@rh-srv1 ~]# ipa-client-install
WARNING: ntpd time&date synchronization service will not be configured as
conflicting service (chronyd) is enabled
Use --force-ntpd option to disable it and force configuration of ntpd

Discovery was successful!
Client hostname: rh-srv1.whittenberg.local
Realm: WHITTENBERG.LOCAL
DNS Domain: whittenberg.local
IPA Server: rh-ipa.whittenberg.local
BaseDN: dc=whittenberg,dc=local

Continue to configure the system with these values? [no]: yes
Skipping synchronizing time with NTP server.
User authorized to enroll computers: admin
Password for admin@WHITTENBERG.LOCAL:
Successfully retrieved CA cert
    Subject:     CN=Certificate Authority,O=WHITTENBERG.LOCAL
    Issuer:      CN=Certificate Authority,O=WHITTENBERG.LOCAL
    Valid From: Fri Oct 21 03:00:04 2016 UTC
    Valid Until: Tue Oct 21 03:00:04 2036 UTC

Enrolled in IPA realm WHITTENBERG.LOCAL
Created /etc/ipa/default.conf
New SSSD config will be created
Configured sudoers in /etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Configured /etc/krb5.conf for IPA realm WHITTENBERG.LOCAL
trying https://rh-ipa.whittenberg.local/ipa/json
Forwarding 'ping' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
Forwarding 'ca_is_enabled' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
Systemwide CA database updated.
Added CA certificates to the default NSS database.
Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub
Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub
Forwarding 'host_mod' to json server 'https://rh-ipa.whittenberg.local/ipa/json'
SSSD enabled
Configured /etc/openldap/ldap.conf
Configured /etc/ssh/ssh_config
Configured /etc/ssh/sshd_config
Configuring whittenberg.local as NIS domain.
Client configuration complete.

Setup the server to auto create home directories for ipa users when first login from the server.

[root@rh-srv1 ~]# authconfig --enablemkhomedir --update
getsebool: SELinux is disabled

Now logout as local user root and back in as ipa user admin.

[root@rh-srv1 ~]# logout

login as: admin
admin@192.168.10.31's password:
Creating home directory for admin.
[admin@rh-srv1 ~]$

[admin@rh-srv1 ~]$ pwd
/home/admin

[admin@rh-srv1 ~]$ whoami
admin

[admin@rh-srv1 ~]$ ping rh-ipa
PING rh-ipa.whittenberg.local (192.168.10.30) 56(84) bytes of data.
64 bytes from rh-ipa.whittenberg.local (192.168.10.30): icmp_seq=1 ttl=64 time=0.084 ms
64 bytes from rh-ipa.whittenberg.local (192.168.10.30): icmp_seq=2 ttl=64 time=0.084 ms
^C
--- rh-ipa.whittenberg.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.084/0.084/0.084/0.000 ms
Now you should be able to ssh into the rh-ipa server without supplying credentials if still login as admin on rh-srv1.

[admin@rh-srv1 ~]$ ssh rh-ipa
Could not chdir to home directory /home/admin: No such file or directory
-bash-4.2$

It worked! Looks like we have home dir issue for ipa user admin on the ipa server though. Lets try and take care of that. First exit back to rh-srv1.

-bash-4.2$ exit
logout
Connection to rh-ipa closed.
[admin@rh-srv1 ~]$

Now ssh to rh-ipa as local user root.

[admin@rh-srv1 ~]$ ssh root@rh-ipa
root@rh-ipa's password:
Last failed login: Fri Oct 21 22:40:19 MST 2016 from 192.168.10.31 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Thu Oct 20 21:19:20 2016 from 192.168.10.5
[root@rh-ipa ~]#

Setup the server to auto create home directories for ipa users when first login from the server.

[root@rh-srv1 ~]# authconfig --enablemkhomedir --update
Now exit back to rh-srv1.

[root@rh-ipa ~]# exit
logout
Connection to rh-ipa closed.
[admin@rh-srv1 ~]$

Now ssh back in to rh-ipa as ipa user admin.

[admin@rh-srv1 ~]$ ssh rh-ipa
Creating home directory for admin.
Last login: Fri Oct 21 22:36:53 2016 from 192.168.10.31
[admin@rh-ipa ~]$ pwd
/home/admin
[admin@rh-ipa ~]$

That solved the problem!!

NOW REPEAT all the steps above in ipa client install section for rh-srv2, of course using the correct IP and hostname for each server.

Thats it folks. Enjoy!

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

Thursday, October 20, 2016

RHCSA Lab setup using CentOS 7

RHCSA Lab setup using CentOS 7

I will using the CentOS-7-x86_64-Everything.iso for the installs in my VMware vSphere Client 6.0 and ESXi 6.0. Each Server gets 2 Cores, 2GB Ram, 25GB disk, and 1 NIC.

Servers names and IP addresses will be:
ipa.whittenberg.local 192.168.10.39
srv1.whittenberg.local 192.168.10.40
srv2.whittenberg.local 192.168.10.41

My Local LAN router is 192.168.10.1 and i will use this for my gateway out and DNS server.

At the CentOS 7 install I set my Network and Hostname.

Click on Network and Hostname on home screen.
Hostname: rh-ipa.whittenberg.local
Click slider to turn ethernet adapter ON. This gives me an IP from my DHCP server from my local LAN router.
Click the Configure button.
[General]
click on Automatically connect to this network
[IPv4 Settings]
Method: Manual
Addresses, click add button
address: 192.168.10.39
netmask: 24
gateway: 192.168.10.1
DNS server: 192.168.10.1
Click Save button
Click Done button

You should now be able to ping 192.168.10.39 from your workstation on the LAN.

Click on Date & Time from the home screen.
Chose your region and city.
Make sure Network Time slider shows ON.
Click on gears next to slider. Time servers should be green. If red then you are unable to reach them.
Click Done button.

Click on Software Selection from home screen.
Select Minimal Install under Base Environment and nothing under Add-Ons.
Click Done button.

Click on Installation Destination from home screen.
Make sure white check mark on your hard disk.
Select I will configure partitioning.
Click Done button.
Click the + button and setup as follows.
/boot = 512MB
swap = 1024MB
/root = 24.5GB
Click Done button.
Accept Changes.

Now click the Begin Installation button.

Supply root password while the installation is taking place.

When finished click the Reboot button.

Putty/SSH into your server from your workstation.

Disable selinux.

cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux
cat /etc/sysconfig/selinux

reboot

Disable and turn off firewalld.

systemctl disable firewalld
systemctl stop firewalld

cat or edit your /etc/hostname file and add the following entree if not there already.

ipa.whittenberg.local

cat or edit your /etc/hosts file and add the following entries if not there already.

192.168.10.39 ipa.whittenberg.local ipa

cat or edit your /etc/resolv.conf file and add the following entries if not there already.

search whittenberg.local
nameserver 192.168.10.1

Install additional packages I need on normal basis.

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git

yum -y groupinstall "Development Tools"

Now update the server.

yum -y update

reboot

**Optional - Install of GNOME Desktop.
yum -y groups install "GNOME Desktop"
yum -y update
systemctl set-default graphical.target
systemctl get-default

reboot and make sure GNOME Desktop starts.
**END Optional - Install of GNOME Desktop.

**Optional - Turn off console screen saver and display all boot messages.
vi /etc/default/grub and on the line starting with GRUB_CMDLINE_LINUX= remove 'rhgb' and 'quiet'. Now add 'consoleblank=0' just before the last " on the line. DO NOT include the ' in the 'consoleblank=0' add.

Now make a new grub.cfg using the line below.

grub2-mkconfig -o /boot/grub2/grub.cfg

If using efi then use this line to make new grub.cfg

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Now when you boot your server you will see all the boot messages.
**END Optional - Turn off console screen saver and display all boot messages.

NOW REPEAT all the steps above for the other two servers, of course using the correct IP and hostname for each server.

Now its time to setup ipa on the ipa server and we will user srv1 as our first client. TO BE CONTINUED.

CentOS 7 clear cache

CentOS 7 clear cache

Clear dentries and inodes.

# sync; echo 2 > /proc/sys/vm/drop_caches

Clear PageCache, dentries and inodes.

# sync; echo 3 > /proc/sys/vm/drop_caches

Wednesday, October 19, 2016

CentOS 7 tigervnc install

CentOS 7 tigervnc install

This will get you basic remote access to your GUI desktop as user root.

# yum -y install tigervnc-server

Now run vncserver and supply the password you will use.

# vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth: file /root/.Xauthority does not exist

New 'server1.example.com:1 (root)' desktop is server1.example.com:1

Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/server1.example.com:1.log

Now from your client run your VNC viewer and use like the following for the server address

192.168.1.15:1

Saturday, October 15, 2016

CentOS 7 kernel upgrade using ELRepo. 4.8.1 as of 10/15/2016

CentOS 7 kernel upgrade using ELRepo. 4.8.1 as of 10/15/2016

AS root install the elrepo public key and RPM

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

Enable the elrepo-kerenel

yum --enablerepo=elrepo-kernel

Install new kerenel

yum install kernel-ml

Reboot and run uname -r to see if on new kernel-ml

# uname -r
3.10.0-327.28.2.el7.x86_64

If still on old kernel, like above, make sure your default menu entry is on the correct entry.

Change default menu entry.

cat /boot/grub2/grub.cfg | grep menuentry

or

cat /boot/efi/EFI/centos/grub.cfg | grep menuentry (if using EFI)

And look for your 4.7 entry. My menu entry is the first one (or 0) so set it to that.

grub2-set-default 0

reboot

Now check version.

# uname -r
4.8.1-1.el7.elrepo.x86_64

If using btrfs remember to upgrade your btrfs-progs to current version as well

First check current version

# btrfs --version
btrfs-progs v3.19.1

Your btrfs-progs should match your kernel.

Uninstall you current btrfs-progs

yum -y remove btrfs-progs

Now install packages for new btrfs-progs

yum install libuuid-devel libattr-devel zlib-devel libacl-devel e2fsprogs-devel libblkid-devel lzo2-devel lzo-devel asciidoc xmlto git
yum -y groupinstall "Development Tools"

Now download, compile, and install new btrfs-progs

cd /root
git clone git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
cd btrfs-progs
./autogen.sh
./configure && make
make install

Now reboot and check versions

# uname -r
4.8.1-1.el7.elrepo.x86_64
# btrfs --version
btrfs-progs v4.8.1

Saturday, October 8, 2016

owncloud 9.1 install on CentOS 7

Fresh install CentOS 7 minimal.

Disable selinux and reboot server.

Update OS and then reboot.

yum -y update
reboot

Run the below command to install MariaDB server and client, httpd server, and PHP. We will use MariaDB to connect ownCloud to its own MySQL database.

yum -x php-mysqlnd -y install mariadb-server mariadb* httpd php* wget

Enable the services.

systemctl enable mariadb
systemctl enable httpd
systemctl start mariadb
systemctl start httpd

Check that these are installed and running.

php -v
systemctl status mariadb
systemctl status httpd

Secure you MySQL server (mariadb). In this example i set the password as mypassword.

mysql_secure_installation
Remove anonymous users? [Y/n] y
... Success!
Disallow root login remotely? [Y/n] n
... skipping.
Remove test database and access to it? [Y/n] y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reload privilege tables now? [Y/n] y
... Success!

Now log into mysql.

mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 10
Server version: 5.5.50-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
+--------------------+
3 rows in set (0.00 sec)

MariaDB [(none)]> quit
Bye

Check the httpd service with your we browser.

http://your_ip_here/

Install owncloud 9.1

Run the following commands as root to trust the repository, install the repo, and install owncloud.

rpm --import https://download.owncloud.org/download/repositories/9.1/CentOS_7/repodata/repomd.xml.key
wget http://download.owncloud.org/download/repositories/9.1/CentOS_7/ce:9.1.repo -O /etc/yum.repos.d/ce:9.1.repo
yum clean expire-cache
yum -y install owncloud-files

Once done with the installation, browse to http://your_ip_here/owncloud/

At the bash shell cli create directory to store your data and change the permissions to that directory.

mkdir /owncloud
chown apache:apache /owncloud
chmod 771 /owncloud

Now back to the web UI of owncloud

Supply admin login and admin password.
Click on Storage & database drop down and supply path for data files, select MySQL/MariaDB and supply user root and root password for your MySQL database and choose owncloud for your database.

Install your clients and enjoy!

Friday, September 16, 2016

Replace bad drive on btrfs 8 drive raid10 array in NAS

Replace bad drive on btrfs 8 drive raid10 array in NAS

My first failed drive in a btrfs file system. Here it goes :)

NAS build overview first.

•    Supermicro C7Z170-OCE Motherboard
•    Intel i5-6600k with Hyper 212 EVO cooler
•    64GB DDR4 Corsair LPX RAM (4x16GB)
•    Samsung 850 pro 120GB SSD for boot and OS
•    Supermicro 8-Port SAS/SATA Card - (AOC-SAS2LP-MV8) 8-channel SAS/SATA adapter with 600MB/s per channel
•    8x WD Red NAS 3TB drives in btrfs raid10 on the Supermicro 8-Port SAS/SATA Card
•    10G-Tek 10Gb dual port CNA card SFP+
•    Fractal Design R5 case with Noctua SSO2 NF-A14 case fans
•    Running CentOS 7 with 4.7 kernel and btrfs-progs v4.7
•    Using Plex Media Server, SAMBA, and NFS (10Gb Ethernet to my ESXi server so I can run images off of the btrfs array over NFS).

Drive 5 (/dev/sdf) went bad. Cluster errors. Ran badblocks check and was spitting tons of errors. All other drives checked out fine. Command to check bad blocks is:

badblocks -v /dev/sd*

With * being the letter of your device. I ran them all at the same time, in order to save time. Commands I used for this are:

badblocks -v /dev/sdb > /tmp/bad-blocks-b.txt &
badblocks -v /dev/sdc > /tmp/bad-blocks-c.txt &
badblocks -v /dev/sdd > /tmp/bad-blocks-d.txt &
badblocks -v /dev/sde > /tmp/bad-blocks-e.txt &
badblocks -v /dev/sdf > /tmp/bad-blocks-f.txt &
badblocks -v /dev/sdg > /tmp/bad-blocks-g.txt &
badblocks -v /dev/sdh > /tmp/bad-blocks-h.txt &
badblocks -v /dev/sdi > /tmp/bad-blocks-i.txt &

Now with all my checks running in the background I can monitor things via the txt files. You can see the processes running using "htop", "ps -ef | grep badblocks", or whatever you like.

I noticed the /tmp/bad-blocks-f.txt growing to over a Gig in size so i killed this process, as i already new drive was bad. I let the others run to completion. Not sure how long this took. It ran over night and I did not try and time it.

FYI The command i used to create the array originally was:

[root@nas /]# mkfs.btrfs -f -m raid10 -d raid10 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

As you can see I used raid10 for the metadata (-m) and the data (-d) and labeled it myraid (-L) using all 8 drives. The -f was to force.

I noticed the bad drive on 9/13/2016 and ordered new drive from Amazon that day. Drive was delivered to me on 9/15/2016. I put the drive in my PC and ran WD Diags on it with extended test. It took about 4 hours to complete. 100% pass.

I don’t have any spare sata ports in my NAS so I’m going to have to remove the drive and replace with new drive. I ran "smartctl -i" on all my drives so I could get serial numbers and make sure I replaced the correct drive before I shut down. My drives are /dev/sd[b-i] (/dev/sda is an SSD I use for boot and OS).

[root@nas ~]# smartctl -i /dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red (AF)
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N0M1P20N
LU WWN Device Id: 5 0014ee 6b0f85183
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Sep 13 14:20:53 2016 MST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

So my serial number for this drive, /dev/sdf, is WMC4N0M1P20N. I shut down the system, found and replaced drive. On boot it got stuck on mounting the btrfs filesystem that I auto mount in /etc/fstab, as it has a missing drive now (The new drive I replaced the old one with is not setup for the array yet so it says its missing). I rebooted in safe mode and remd out the line in /etc/fstab for this mount. That line was:

UUID=1ec4f641-74a8-466e-89cc-e687672aaaea   /myraid   btrfs defaults,nodatacow,noatime,x-systemd.device-timeout=0 0 0

So I just put a # in front of it, saved and exit. Reboot system. System up now. Now to work with the btrfs file system I have to get it mounted, so I mount it in degraded mode, as it is degraded with the missing disk. Command I used for that is:

mount -t btrfs -o degraded UUID=1ec4f641-74a8-466e-89cc-e687672aaaea /myraid

BTW i use the UUID for my mounts. You can get this from running "btrfs fi show".

I now inspect the filesystem with:

[root@nas ~]# btrfs fi show
Label: 'myraid' uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 8 FS bytes used 788.93GiB
        devid    1 size 2.73TiB used 200.27GiB path /dev/sdb
        devid    2 size 2.73TiB used 200.27GiB path /dev/sdc
        devid    3 size 2.73TiB used 200.27GiB path /dev/sdd
        devid    4 size 2.73TiB used 200.27GiB path /dev/sde
        devid    6 size 2.73TiB used 200.27GiB path /dev/sdg
        devid    7 size 2.73TiB used 200.27GiB path /dev/sdh
        devid    8 size 2.73TiB used 200.27GiB path /dev/sdi
        *** Some devices missing


As you can see there is no devid 5, /dev/sdf. Now I can tell btrfs to delete the missing device with:

[root@nas ~]# btrfs device delete missing /myraid/

It goes through a lengthy process of relocating blocks from /dev/sdg. Raid10 is striped mirrors so it pairs them like /dev/sdb with /dev/sdc, /dev/sdd with /dev/sde, /dev/sdf with /dev/sdg, etc.

Sep 16 17:04:44 nas kernel: BTRFS info (device sdg): relocating block group 2407559856128 flags 65
Sep 16 17:04:52 nas kernel: BTRFS info (device sdg): found 43 extents
Sep 16 17:05:02 nas kernel: BTRFS info (device sdg): found 43 extents
Sep 16 17:05:03 nas kernel: BTRFS info (device sdg): relocating block group 2394674954240 flags 65
Sep 16 17:05:10 nas kernel: BTRFS info (device sdg): found 38 extents
Sep 16 17:05:20 nas kernel: BTRFS info (device sdg): found 38 extents
Sep 16 17:05:21 nas kernel: BTRFS info (device sdg): relocating block group 2738742099968 flags 65
Sep 16 17:05:28 nas kernel: BTRFS info (device sdg): found 32 extents
Sep 16 17:05:38 nas kernel: BTRFS info (device sdg): found 32 extents
.....

The lines above get displayed on the console. If you are ssh into your system you can also see them in /var/log/messages.

Now I wait for the process to complete. Start time is 5:04:44PM AZ time on 9/16/2016. (Insert jeopardy theme song here).

It’s now 5:52:18PM AZ time. More jeopardy theme music.

6:05:24PM and it finished. Let’s inspect.

[root@nas log]# btrfs fi show
Label: 'myraid' uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 7 FS bytes used 788.83GiB
        devid    1 size 2.73TiB used 226.34GiB path /dev/sdb
        devid    2 size 2.73TiB used 225.38GiB path /dev/sdc
        devid    3 size 2.73TiB used 226.03GiB path /dev/sdd
        devid    4 size 2.73TiB used 225.38GiB path /dev/sde
        devid    6 size 2.73TiB used 226.38GiB path /dev/sdg
        devid    7 size 2.73TiB used 225.38GiB path /dev/sdh
        devid    8 size 2.73TiB used 225.38GiB path /dev/sdi


SWEET! now more missing drive. Notice how the data on each of the drives is more than when I ran the command with the missing drive.

Now let’s take out the rem from the /etc/fstab file and reboot the system. Fingers crossed :)

System booted fine but my space on the array seems a little off. Before this started I had 11TB of disk space show up when I ran "df -h". Now its:

[root@nas ~]# df -h
Filesystem                   Size Used Avail Use% Mounted on
devtmpfs                      32G     0   32G   0% /dev
tmpfs                         32G 4.0K   32G   1% /dev/shm
tmpfs                         32G 8.9M   32G   1% /run
tmpfs                         32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/centos_nas-root   50G 4.4G   46G   9% /
/dev/mapper/centos_nas-home   57G   33M   57G   1% /home
/dev/sda2                    497M 116M 381M 24% /boot
/dev/sda1                    200M 9.5M 191M   5% /boot/efi
/dev/sdb                     9.6T 789G 5.1T 14% /myraid
tmpfs                        6.3G     0 6.3G   0% /run/user/0

Yes, that’s right. 9.6TB now. So I am truly not in a raid10 anymore as my data seems to be striped across all my drives and not in mirrored pairs. Makes sense as don’t have even number of drives now. hmmmmmm.... Not sure how I can get this back into striped mirrors.

It still shows in raid10 though.

[root@nas ~]# btrfs fi df -h /myraid/
Data, RAID10: total=789.00GiB, used=788.83GiB
System, RAID10: total=96.00MiB, used=96.00KiB
Metadata, RAID10: total=1.03GiB, used=6.50MiB
GlobalReserve, single: total=16.00MiB, used=0.00B

First inspect filesystem.

[root@nas ~]# btrfs fi show
Label: 'myraid' uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 7 FS bytes used 788.83GiB
        devid    1 size 2.73TiB used 226.34GiB path /dev/sdb
        devid    2 size 2.73TiB used 225.38GiB path /dev/sdc
        devid    3 size 2.73TiB used 226.03GiB path /dev/sdd
        devid    4 size 2.73TiB used 225.38GiB path /dev/sde
        devid    6 size 2.73TiB used 226.38GiB path /dev/sdg
        devid    7 size 2.73TiB used 225.38GiB path /dev/sdh
        devid    8 size 2.73TiB used 225.38GiB path /dev/sdi

Now let’s add the new drive back in.

[root@nas ~]# btrfs dev add -f /dev/sdf /myraid/

That only took a second.

Inspect filesystem.

[root@nas ~]# btrfs fi show
Label: 'myraid' uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 8 FS bytes used 788.83GiB
        devid    1 size 2.73TiB used 226.34GiB path /dev/sdb
        devid    2 size 2.73TiB used 225.38GiB path /dev/sdc
        devid    3 size 2.73TiB used 226.03GiB path /dev/sdd
        devid    4 size 2.73TiB used 225.38GiB path /dev/sde
        devid    6 size 2.73TiB used 226.38GiB path /dev/sdg
        devid    7 size 2.73TiB used 225.38GiB path /dev/sdh
        devid    8 size 2.73TiB used 225.38GiB path /dev/sdi
        devid    9 size 2.73TiB used 0.00B path /dev/sdf

Well I have 8 drives again. That’s a start :) But no data on it, of course.

I also see my devid for the new drive is 9, and no longer shows a 5. Hmmmmmm....

Let’s balance out the filesystem now and see what it looks like when done.

[root@nas ~]# btrfs bala start -v /myraid/

You can monitor the progress with:

[root@nas ~]# btrfs bal status -v /myraid/
Balance on '/myraid/' is running
18 out of about 265 chunks balanced (19 considered), 93% left
Dumping filters: flags 0x7, state 0x1, force is off
DATA (flags 0x0): balancing
METADATA (flags 0x0): balancing
SYSTEM (flags 0x0): balancing

Start time 6:42:36PM AZ time. Is that Jeopardy music I here. Time to binge on walking dead season six for a bit.

Oh ya. Remembered to turn head parking off on the new drive as well using tool idle3ctl. Command is:

Get current setting on /dev/sdf

[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdf
Idle3 timer set to 138 (0x8a)

Turn parking off

[root@nas idle3-tools-0.9.1]# ./idle3ctl -d /dev/sdf
Idle3 timer disabled
Please power cycle your drive off and on for the new setting to be taken into account. A reboot will not be enough!

Recheck setting

[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdf
Idle3 timer is disabled

I will power cycle later, when the balance is done running. I will then recheck the setting to verify.

Aslo check all firmware versions

[root@nas ~]# ./wd5741x64
WD5741 Version 1
Update Drive
Copyright (C) 2013 Western Digital Corporation

-Dn   Model String           Serial Number     Firmware
-D0   Samsung SSD 850 PRO 128GB   S1SMNSAG301480T   EXM02B6Q
-D1   WDC WD30EFRX-68EUZN0   WD-WMC4N0J0YT1V   82.00A82
-D2   WDC WD30EFRX-68EUZN0   WD-WMC4N0J2L138   82.00A82
-D3   WDC WD30EFRX-68EUZN0   WD-WCC4N2FJRTU9   82.00A82
-D4   WDC WD30EFRX-68EUZN0   WD-WCC4N4SSDRFN   82.00A82
-D5   WDC WD30EFRX-68EUZN0   WD-WCC4N1VYZH52   82.00A82
-D6   WDC WD30EFRX-68EUZN0   WD-WMC4N0M57KEY   82.00A82
-D7   WDC WD30EFRX-68EUZN0   WD-WCC4N5YF2Z2Y   82.00A82
-D8   WDC WD30EFRX-68EUZN0   WD-WCC4N5CJ6H8U   82.00A82
[root@nas ~]#

Firmware is also displayed running "smartctl -i" we did earlier to get all the drives serial numbers.

So at 7:46:18PM it finished. Lets inspect.

[root@nas ~]# btrfs fi sho
Label: 'myraid' uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 8 FS bytes used 788.91GiB
        devid    1 size 2.73TiB used 199.27GiB path /dev/sdb
        devid    2 size 2.73TiB used 199.27GiB path /dev/sdc
        devid    3 size 2.73TiB used 199.27GiB path /dev/sdd
        devid    4 size 2.73TiB used 199.27GiB path /dev/sde
        devid    6 size 2.73TiB used 199.27GiB path /dev/sdg
        devid    7 size 2.73TiB used 199.27GiB path /dev/sdh
        devid    8 size 2.73TiB used 199.27GiB path /dev/sdi
        devid    9 size 2.73TiB used 199.27GiB path /dev/sdf
So my data is even across all eight drives. It also shows 1 gig less on each drive but the before drive replacement.

[root@nas ~]# df -h
Filesystem                   Size Used Avail Use% Mounted on
devtmpfs                      32G     0   32G   0% /dev
tmpfs                         32G 4.0K   32G   1% /dev/shm
tmpfs                         32G 8.9M   32G   1% /run
tmpfs                         32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/centos_nas-root   50G 4.4G   46G   9% /
/dev/mapper/centos_nas-home   57G   33M   57G   1% /home
/dev/sda2                    497M 116M 381M 24% /boot
/dev/sda1                    200M 9.5M 191M   5% /boot/efi
/dev/sdb                      11T 789G   11T   8% /myraid
tmpfs                        6.3G     0 6.3G   0% /run/user/0

I also have my 11TB showing up on df -h.

[root@nas ~]# btrfs fi df /myraid/
Data, RAID10: total=796.00GiB, used=788.90GiB
System, RAID10: total=64.00MiB, used=112.00KiB
Metadata, RAID10: total=1.00GiB, used=6.34MiB
GlobalReserve, single: total=16.00MiB, used=0.00B
looks like it’s in raid10 mode still. One thing I did notice was on the balance it was using /dev/sdc, and not /dev/sdg. Remember it used /dev/sdg for relocating blocks when i removed /dev/sdf. But when it balanced after the new drive in /dev/sdf it used /dev/sdc for relocating blocks. See below:

Sep 16 19:41:58 nas kernel: BTRFS info (device sdc): relocating block group 2897790107648 flags 65
Sep 16 19:42:07 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:13 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:13 nas kernel: BTRFS info (device sdc): relocating block group 2894568882176 flags 65
Sep 16 19:42:20 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:27 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:27 nas kernel: BTRFS info (device sdc): relocating block group 2891347656704 flags 65
Sep 16 19:42:35 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:41 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:41 nas kernel: BTRFS info (device sdc): relocating block group 2888126431232 flags 65
Sep 16 19:42:48 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:55 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:42:55 nas kernel: BTRFS info (device sdc): relocating block group 2884905205760 flags 65
Sep 16 19:43:02 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:43:08 nas kernel: BTRFS info (device sdc): found 24 extents
Sep 16 19:43:09 nas kernel: BTRFS info (device sdc): relocating block group 2881683980288 flags 65
.....

Let’s run some bench marks on this baby now. Woop Woop! Oh lets power cycle and look at head parking again first.

Powered back up and looks good on head parking.

[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdb
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdc
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdd
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sde
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdf
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdg
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdh
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl -g /dev/sdi
Idle3 timer is disabled

Let’s do a scrub before the benchmarks just to be sure.

[root@nas idle3-tools-0.9.1]# btrfs scrub start /myraid/
scrub started on /myraid/, fsid 1ec4f641-74a8-466e-89cc-e687672aaaea (pid=2442)

You can check status of the scrub with:

[root@nas idle3-tools-0.9.1]# btrfs scrub status -d /myraid/
scrub status for 1ec4f641-74a8-466e-89cc-e687672aaaea
scrub device /dev/sdb (id 1) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.55GiB with 0 errors
scrub device /dev/sdc (id 2) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 8.45GiB with 0 errors
scrub device /dev/sdd (id 3) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 8.36GiB with 0 errors
scrub device /dev/sde (id 4) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.59GiB with 0 errors
scrub device /dev/sdg (id 6) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.71GiB with 0 errors
scrub device /dev/sdh (id 7) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.64GiB with 0 errors
scrub device /dev/sdi (id 8) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.56GiB with 0 errors
scrub device /dev/sdf (id 9) status
        scrub started at Fri Sep 16 19:59:37 2016, running for 00:01:05
        total bytes scrubbed: 9.55GiB with 0 errors

Now we wait again. I HATE WAITING!!

Using iotop while scrubbing. Each drive is showing about 150MB read. With a total showing 1,200MB. 150 x 8 = 1,200. These WD drives only get a max of 150MB so it looks good so far. Since I have about 790GB of data I will need to scrub, double that, as it is a mirrored setup (striped mirrors to be exact). So I’m looking at about 1.58TB of data to scrub as I will scrub both copies.

Finished the scrub.

[root@nas idle3-tools-0.9.1]# btrfs scrub status /myraid/
scrub status for 1ec4f641-74a8-466e-89cc-e687672aaaea
        scrub started at Fri Sep 16 19:59:37 2016 and finished after 00:25:29
        total bytes scrubbed: 1.54TiB with 0 errors
Zero Errors. Hell ya!

Ok now for benchmarks. NAS is back in its home with dual 1Gb Ethernet and dual 10Gb Ethernet connections now. One of the 10Gb connections is to my PC. My PC has a Samsung 950 pro M.2 SSD drive that has reads of over 2,500MB and writes over 1,500MB. I have benchmarked this and it is true. So with a 10Gb Ethernet connection, highly tuned (and mine is) I can get 1,200MB transfers across the wire. But let’s be real. Even with 8 striped mirrors WD Red NAS drives I will never get that. With each drive at 150MB x4 = 600MB. I say times 4 as the other four drives are mirrored. So for a non cached read from my btrfs 8 drive raid10 array across the 10GB Ethernet connection to my 950 pro SSD I should get a max about 600MB. Drum roll please...... 570MB transfer with a 10GB file across the wire. Sweet!!! This is over a samba share using windows explorer so I got a bit of overhead on that so 570MB is pretty darn good. Who else has a home NAS and network that allows them that kind of speed?

Now that the file I just transferred is cashed in RAM lets try the transfer again. Yikes Scoob! 1,150MB transfer speed. Basically maxing out a 10Gb Ethernet connection, considering a little smb and windows overhead.

I also have Samsung 850 pro SSD 120GB in the NAS i use for boot and OS, so it has some free space to say the least. Let’s try a transfer from btrfs raid10 array to the SSD in the NAS. This SSD has a mx Write of just over 500MB. 520MB I think it is. So let’s reboot the server so the file is not cached and try that test.

My 10GB test file is on the array in /myraid mount. file name is 10gtest.file
I wrote a script that gives the date and time, then copies the file from the array to the SSD drive, then gives the date and time again.

#!/bin/bash
date
cp /myraid/10gtest.file /root/10gtest.file
date

Now lets run the script.

[root@nas ~]# ./test.sh
Fri Sep 16 21:00:05 MST 2016
Fri Sep 16 21:00:30 MST 2016

25 seconds nets us about 400MB. was looking for more than that. Hmmmmm.....

Now again with file cached.

[root@nas ~]# ./test.sh
Fri Sep 16 21:07:34 MST 2016
Fri Sep 16 21:07:38 MST 2016

5 seconds nets us about 2,500MB. Ya memory and the system bus is really fast. Makes me smile.

Now for some write test. reboot NAS so files aren’t cached.

From PC to NAS was 1,200MB. OK, looks like it is going to RAM before making it to the array. I can live with That :) Maxing out the 10Gb Ethernet is always good.

Well guys and gals my time here is done. I’m calling this a success. I will be researching the slow SSD drive in my NAS. It is about three years old now has been in a few systems of mine for different purposes. Could be time for a new one. I will also be looking into the devid numbering. Kind of bummed it did not go back as 5, but rather there is no 5 and created a new one called 9. Still 8 drives just skipped a devid. I also need to figure out if I’m still a raid10 even though it used /dev/sdg when I removed /dev/sdf, that part makes since, but used /dev/sdc after I added the new drive and balanced.

Anyone have any ideas to my 3 questions? Did I do something wrong?

Until next time, Peace!

Broadsoft, Adtran, and Centos

Why?

Search This Blog

Tuesday, October 25, 2016

CentOS 7 sshd server to server with no password using trusted keys

Saturday, October 22, 2016

CentOS 7 rescue, backup, and restore using ReaR

Friday, October 21, 2016

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

RHCSA Lab setup using CentOS 7 with ipa server and ipa client

Thursday, October 20, 2016

RHCSA Lab setup using CentOS 7

CentOS 7 clear cache

Wednesday, October 19, 2016

CentOS 7 tigervnc install

Saturday, October 15, 2016

CentOS 7 kernel upgrade using ELRepo. 4.8.1 as of 10/15/2016

Saturday, October 8, 2016

owncloud 9.1 install on CentOS 7

Friday, September 16, 2016

Replace bad drive on btrfs 8 drive raid10 array in NAS