Why?


Search This Blog

Saturday, October 28, 2017

Docker on CentOS 7

This is copy of:
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-centos-7

Great to the point article that got the job done for me. Thanks!

How To Install and Use Docker on CentOS 7

UpdatedNovember 2, 2016 163.1k views Docker CentOS

Introduction

Docker is an application that makes it simple and easy to run application processes in a container, which are like virtual machines, only more portable, more resource-friendly, and more dependent on the host operating system. For a detailed introduction to the different components of a Docker container, check out The Docker Ecosystem: An Introduction to Common Components.
There are two methods for installing Docker on CentOS 7. One method involves installing it on an existing installation of the operating system. The other involves spinning up a server with a tool called Docker Machine that auto-installs Docker on it.
In this tutorial, you'll learn how to install and use it on an existing installation of CentOS 7.

Prerequisites

Note: Docker requires a 64-bit version of CentOS 7 as well as a kernel version equal to or greater than 3.10. The default 64-bit CentOS 7 Droplet meets these requirements.
All the commands in this tutorial should be run as a non-root user. If root access is required for the command, it will be preceded by sudo. Initial Setup Guide for CentOS 7 explains how to add users and give them sudo access.

Step 1 — Installing Docker

The Docker installation package available in the official CentOS 7 repository may not be the latest version. To get the latest and greatest version, install Docker from the official Docker repository. This section shows you how to do just that.
But first, let's update the package database:
  • sudo yum check-update
Now run this command. It will add the official Docker repository, download the latest version of Docker, and install it:
  • curl -fsSL https://get.docker.com/ | sh
After installation has completed, start the Docker daemon:
  • sudo systemctl start docker
Verify that it's running:
  • sudo systemctl status docker
The output should be similar to the following, showing that the service is active and running:
Output
● docker.service - Docker Application Container Engine Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2016-05-01 06:53:52 CDT; 1 weeks 3 days ago Docs: https://docs.docker.com Main PID: 749 (docker)
Lastly, make sure it starts at every server reboot:
  • sudo systemctl enable docker
Installing Docker now gives you not just the Docker service (daemon) but also the docker command line utility, or the Docker client. We'll explore how to use the docker command later in this tutorial.

Step 2 — Executing Docker Command Without Sudo (Optional)

By default, running the docker command requires root privileges — that is, you have to prefix the command with sudo. It can also be run by a user in the docker group, which is automatically created during the installation of Docker. If you attempt to run the docker command without prefixing it with sudo or without being in the docker group, you'll get an output like this:
Output
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. See 'docker run --help'.
If you want to avoid typing sudo whenever you run the docker command, add your username to the docker group:
  • sudo usermod -aG docker $(whoami)
You will need to log out of the Droplet and back in as the same user to enable this change.
If you need to add a user to the docker group that you're not logged in as, declare that username explicitly using:
  • sudo usermod -aG docker username
The rest of this article assumes you are running the docker command as a user in the docker user group. If you choose not to, please prepend the commands with sudo.

Step 3 — Using the Docker Command

With Docker installed and working, now's the time to become familiar with the command line utility. Using docker consists of passing it a chain of options and subcommands followed by arguments. The syntax takes this form:
  • docker [option] [command] [arguments]
To view all available subcommands, type:
  • docker
As of Docker 1.11.1, the complete list of available subcommands includes:
Output
attach Attach to a running container build Build an image from a Dockerfile commit Create a new image from a container's changes cp Copy files/folders between a container and the local filesystem create Create a new container diff Inspect changes on a container's filesystem events Get real time events from the server exec Run a command in a running container export Export a container's filesystem as a tar archive history Show the history of an image images List images import Import the contents from a tarball to create a filesystem image info Display system-wide information inspect Return low-level information on a container or image kill Kill a running container load Load an image from a tar archive or STDIN login Log in to a Docker registry logout Log out from a Docker registry logs Fetch the logs of a container network Manage Docker networks pause Pause all processes within a container port List port mappings or a specific mapping for the CONTAINER ps List containers pull Pull an image or a repository from a registry push Push an image or a repository to a registry rename Rename a container restart Restart a container rm Remove one or more containers rmi Remove one or more images run Run a command in a new container save Save one or more images to a tar archive search Search the Docker Hub for images start Start one or more stopped containers stats Display a live stream of container(s) resource usage statistics stop Stop a running container tag Tag an image into a repository top Display the running processes of a container unpause Unpause all processes within a container update Update configuration of one or more containers version Show the Docker version information volume Manage Docker volumes wait Block until a container stops, then print its exit code
To view the switches available to a specific command, type:
  • docker docker-subcommand --help
To view system-wide information, use:
  • docker info

Step 4 — Working with Docker Images

Docker containers are run from Docker images. By default, it pulls these images from Docker Hub, a Docker registry managed by Docker, the company behind the Docker project. Anybody can build and host their Docker images on Docker Hub, so most applications and Linux distributions you'll need to run Docker containers have images that are hosted on Docker Hub.
To check whether you can access and download images from Docker Hub, type:
  • docker run hello-world
The output, which should include the following, should indicate that Docker in working correctly:
Output
Hello from Docker. This message shows that your installation appears to be working correctly. ...
You can search for images available on Docker Hub by using the docker command with the search subcommand. For example, to search for the CentOS image, type:
  • docker search centos
The script will crawl Docker Hub and return a listing of all images whose name match the search string. In this case, the output will be similar to this:
Output
NAME DESCRIPTION STARS OFFICIAL AUTOMATED centos The official build of CentOS. 2224 [OK] jdeathe/centos-ssh CentOS-6 6.7 x86_64 / CentOS-7 7.2.1511 x8... 22 [OK] jdeathe/centos-ssh-apache-php CentOS-6 6.7 x86_64 / Apache / PHP / PHP M... 17 [OK] million12/centos-supervisor Base CentOS-7 with supervisord launcher, h... 11 [OK] nimmis/java-centos This is docker images of CentOS 7 with dif... 10 [OK] torusware/speedus-centos Always updated official CentOS docker imag... 8 [OK] nickistre/centos-lamp LAMP on centos setup 3 [OK] ...
In the OFFICIAL column, OK indicates an image built and supported by the company behind the project. Once you've identifed the image that you would like to use, you can download it to your computer using the pull subcommand, like so:
  • docker pull centos
After an image has been downloaded, you may then run a container using the downloaded image with the run subcommand. If an image has not been downloaded when docker is executed with the run subcommand, the Docker client will first download the image, then run a container using it:
  • docker run centos
To see the images that have been downloaded to your computer, type:
  • docker images
The output should look similar to the following:
[secondary_lable Output]
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              latest              778a53015523        5 weeks ago         196.7 MB
hello-world         latest              94df4f0ce8a4        2 weeks ago         967 B
As you'll see later in this tutorial, images that you use to run containers can be modified and used to generate new images, which may then be uploaded (pushed is the technical term) to Docker Hub or other Docker registries.

Step 5 — Running a Docker Container

The hello-world container you ran in the previous step is an example of a container that runs and exits, after emitting a test message. Containers, however, can be much more useful than that, and they can be interactive. After all, they are similar to virtual machines, only more resource-friendly.
As an example, let's run a container using the latest image of CentOS. The combination of the -i and -t switches gives you interactive shell access into the container:
  • docker run -it centos
Your command prompt should change to reflect the fact that you're now working inside the container and should take this form:
Output
[root@59839a1b7de2 /]#
Important: Note the container id in the command prompt. In the above example, it is 59839a1b7de2.
Now you may run any command inside the container. For example, let's install MariaDB server in the running container. No need to prefix any command with sudo, because you're operating inside the container with root privileges:
  • yum install mariadb-server

Step 6 — Committing Changes in a Container to a Docker Image

When you start up a Docker image, you can create, modify, and delete files just like you can with a virtual machine. The changes that you make will only apply to that container. You can start and stop it, but once you destroy it with the docker rm command, the changes will be lost for good.
This section shows you how to save the state of a container as a new Docker image.
After installing MariaDB server inside the CentOS container, you now have a container running off an image, but the container is different from the image you used to create it.
To save the state of the container as a new image, first exit from it:
  • exit
Then commit the changes to a new Docker image instance using the following command. The -m switch is for the commit message that helps you and others know what changes you made, while -a is used to specify the author. The container ID is the one you noted earlier in the tutorial when you started the interactive docker session. Unless you created additional repositories on Docker Hub, the repository is usually your Docker Hub username:
  • docker commit -m "What did you do to the image" -a "Author Name" container-id repository/new_image_name
For example:
  • docker commit -m "added mariadb-server" -a "Sunday Ogwu-Chinuwa" 59839a1b7de2 finid/centos-mariadb
Note: When you commit an image, the new image is saved locally, that is, on your computer. Later in this tutorial, you'll learn how to push an image to a Docker registry like Docker Hub so that it may be assessed and used by you and others.
After that operation has completed, listing the Docker images now on your computer should show the new image, as well as the old one that it was derived from:
  • docker images
The output should be of this sort:
Output
REPOSITORY TAG IMAGE ID CREATED SIZE finid/centos-mariadb latest 23390430ec73 6 seconds ago 424.6 MB centos latest 778a53015523 5 weeks ago 196.7 MB hello-world latest 94df4f0ce8a4 2 weeks ago 967 B
In the above example, centos-mariadb is the new image, which was derived from the existing CentOS image from Docker Hub. The size difference reflects the changes that were made. And in this example, the change was that MariaDB server was installed. So next time you need to run a container using CentOS with MariaDB server pre-installed, you can just use the new image. Images may also be built from what's called a Dockerfile. But that's a very involved process that's well outside the scope of this article. We'll explore that in a future article.

Step 7 — Listing Docker Containers

After using Docker for a while, you'll have many active (running) and inactive containers on your computer. To view the active ones, use:
  • docker ps
You will see output similar to the following:
Output
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f7c79cc556dd centos "/bin/bash" 3 hours ago Up 3 hours silly_spence
To view all containers — active and inactive, pass it the -a switch:
  • docker ps -a
To view the latest container you created, pass it the -l switch:
  • docker ps -l
Stopping a running or active container is as simple as typing:
  • docker stop container-id
The container-id can be found in the output from the docker ps command.

Step 8 — Pushing Docker Images to a Docker Repository

The next logical step after creating a new image from an existing image is to share it with a select few of your friends, the whole world on Docker Hub, or other Docker registry that you have access to. To push an image to Docker Hub or any other Docker registry, you must have an account there.
This section shows you how to push a Docker image to Docker Hub.
To create an account on Docker Hub, register at Docker Hub. Afterwards, to push your image, first log into Docker Hub. You'll be prompted to authenticate:
  • docker login -u docker-registry-username
If you specified the correct password, authentication should succeed. Then you may push your own image using:
  • docker push docker-registry-username/docker-image-name
It will take sometime to complete, and when completed, the output will be of this sort:
Output
The push refers to a repository [docker.io/finid/centos-mariadb] 670194edfaf5: Pushed 5f70bf18a086: Mounted from library/centos 6a6c96337be1: Mounted from library/centos ...
After pushing an image to a registry, it should be listed on your account's dashboard, like that show in the image below.
Docker image listing on Docker Hub
If a push attempt results in an error of this sort, then you likely did not log in:
Output
The push refers to a repository [docker.io/finid/centos-mariadb] e3fbbfb44187: Preparing 5f70bf18a086: Preparing a3b5c80a4eba: Preparing 7f18b442972b: Preparing 3ce512daaf78: Preparing 7aae4540b42d: Waiting unauthorized: authentication required
Log in, then repeat the push attempt.

Conclusion

There's a whole lot more to Docker than has been given in this article, but this should be enough to getting you started working with it on CentOS 7. Like most open source projects, Docker is built from a fast-developing codebase, so make a habit of visiting the project's blog page for the latest information.
Also check out the other Docker tutorials in the DO Community.

Sunday, March 12, 2017

Bad luck with WD Red NAS 3TB drives

3/12/2017

I have had 4 of 8 drives go bad in the last year and half. Oldest drive is 2 years old. Model numbers for all the drives are WD30EFRX-68EUZN0. I am waiting on an RMA to come back, and another went bad tonight. Lucky I have extra drives and am replacing tonight with another new WD Red NAS 3TB drive i had for just this purpose.

I have decided to not get anymore of these drives, other than the RMA I am waiting on now, and the RMA I will do after I replace the current bad drive. I have ordered a single HGST Desktsar NAS 3TB drive to look at. If this will fit my needs I will replace all drives in my NAS with these.

I want to look at the heat, noise, and see if i can get head parking off on the HGST drives. I will also measure performance, but I am sure the 7200 HGST will be faster then the 5400 WD Red.

Just an FYI to everyone. Be careful with these WD Red NAS 3TB drives.

UPDATE 3/13/2017

So i got the drive out of the NAS. Home built  on Centos 7.3, kernel 4.10, btrfs-progs 4.9. Took a bit to get the drive removed from the array (wait for data to be moved across the other 7 drives) and then add the replacement back in (wait for data to be balanced back across all 8 drives again).

I then took the "bad drive" and put it in my Windows workstation. I ran the WD Data Lifeguard extended test and then ran the extended test from Aomei Partition Assistant Pro. Both of these test show I have no bad sectors. The drive seems to be good as new. To make long story short I think i have a bad fan cable from my SAS/SATA controller to the drives. I have two ports on controller that controls 4 SATA drives each. I reseated HBA card and cables, and power connectors on the drives, and ran some test and scrubs. all seems good with new drive. I then put the side panels back on the case and place it back in its home. And within an hour i start getting more errors on the same device id, only with the brand new drive on that id. So i pull the server back out, remove side panels, and reboot. Over 24 hours later I don't have another error. It maybe that the side panels on the case are moving the cables enough for the bad cable to cause the errors. So i have two new fan cables coming and should have them installed tomorrow night. I still have the new HGST drive coming because I really want to check this out. If they are not to noisy and run to hot, I may just replace them all anyway. I better see some latency and IO improvements before that happens. At $150.00 per drive x8..... Well its not cheap to me.

Friday, March 3, 2017

BTRFS Drive errors from 2017-02-26

BTRFS Drive errors from 2017-02-26

Was not getting failed scrubs but seeing some errors.

This is 8x 3TB WD Red drives in raid10 on a Supermicro AOC-SAS2LP-MV8 Add-on Card, 8-Channel SAS/SATA Adapter with 600MB/s per Channel in a PCIE x16 slot running at x8 on a Supermicro ATX DDR4 LGA 1151 C7Z170-OCE-O Motherboard with 64GB DDR4 RAM (4x 16GB sticks).

FYI Everything was done online unless otherwise stated.

What file system looks like in btrfs.

[root@nas ~]# btrfs fi show
Label: 'myraid'  uuid: 1ec4f641-74a8-466e-89cc-e687672aaaea
        Total devices 8 FS bytes used 1.16TiB
        devid    1 size 2.73TiB used 301.53GiB path /dev/sdb
        devid    2 size 2.73TiB used 301.53GiB path /dev/sdc
        devid    3 size 2.73TiB used 301.53GiB path /dev/sdd
        devid    4 size 2.73TiB used 301.53GiB path /dev/sde
        devid    6 size 2.73TiB used 301.53GiB path /dev/sdg
        devid    7 size 2.73TiB used 301.53GiB path /dev/sdh
        devid    8 size 2.73TiB used 301.53GiB path /dev/sdi
        devid    9 size 2.73TiB used 301.53GiB path /dev/sdf


Take a look at device stats.

[root@nas ~]# /usr/local/bin/btrfs device stats /myraid/
[/dev/sdb].write_io_errs   0
[/dev/sdb].read_io_errs    0
[/dev/sdb].flush_io_errs   0
[/dev/sdb].corruption_errs 0
[/dev/sdb].generation_errs 0
[/dev/sdc].write_io_errs   0
[/dev/sdc].read_io_errs    0
[/dev/sdc].flush_io_errs   0
[/dev/sdc].corruption_errs 0
[/dev/sdc].generation_errs 0
[/dev/sdd].write_io_errs   0
[/dev/sdd].read_io_errs    0
[/dev/sdd].flush_io_errs   0
[/dev/sdd].corruption_errs 0
[/dev/sdd].generation_errs 0
[/dev/sde].write_io_errs   0
[/dev/sde].read_io_errs    44
[/dev/sde].flush_io_errs   0
[/dev/sde].corruption_errs 0
[/dev/sde].generation_errs 0
[/dev/sdg].write_io_errs   0
[/dev/sdg].read_io_errs    0
[/dev/sdg].flush_io_errs   0
[/dev/sdg].corruption_errs 0
[/dev/sdg].generation_errs 0
[/dev/sdh].write_io_errs   0
[/dev/sdh].read_io_errs    0
[/dev/sdh].flush_io_errs   0
[/dev/sdh].corruption_errs 0
[/dev/sdh].generation_errs 0
[/dev/sdi].write_io_errs   0
[/dev/sdi].read_io_errs    0
[/dev/sdi].flush_io_errs   0
[/dev/sdi].corruption_errs 0
[/dev/sdi].generation_errs 0
[/dev/sdf].write_io_errs   0
[/dev/sdf].read_io_errs    0
[/dev/sdf].flush_io_errs   0
[/dev/sdf].corruption_errs 0
[/dev/sdf].generation_errs 0


Run extended smartcl test.

[root@nas ~]# smartctl -t long /dev/sdb
[root@nas ~]# smartctl -t long /dev/sdc
[root@nas ~]# smartctl -t long /dev/sdd
[root@nas ~]# smartctl -t long /dev/sde
[root@nas ~]# smartctl -t long /dev/sdf
[root@nas ~]# smartctl -t long /dev/sdg
[root@nas ~]# smartctl -t long /dev/sdh
[root@nas ~]# smartctl -t long /dev/sdi


I waited an hour then reviewed the results.

[root@nas ~]# smartctl -l selftest /dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12660         -
# 2  Extended offline    Completed without error       00%      8916         -
# 3  Short offline       Completed without error       00%      6097         -
# 4  Extended offline    Completed without error       00%      4288         -
# 5  Short offline       Completed without error       00%      4245         -
# 6  Short offline       Completed without error       00%      4242         -
# 7  Short offline       Interrupted (host reset)      50%      4241         -
# 8  Short offline       Completed without error       00%      4172         -
# 9  Short offline       Completed without error       00%      4109         -

[root@nas ~]# smartctl -l selftest /dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     12660         -
# 2  Extended offline    Completed without error       00%      8916         -
# 3  Short offline       Completed without error       00%      6096         -
# 4  Extended offline    Completed without error       00%      4288         -
# 5  Short offline       Completed without error       00%      4109         -

[root@nas ~]# smartctl -l selftest /dev/sdd
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     13003         -
# 2  Extended offline    Completed without error       00%      9260         -
# 3  Short offline       Completed without error       00%      6440         -
# 4  Extended offline    Completed without error       00%      4632         -
# 5  Short offline       Completed without error       00%      4452         -
# 6  Short offline       Completed without error       00%         0         -

[root@nas ~]# smartctl -l selftest /dev/sde
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      8452         339784376
# 2  Extended offline    Completed without error       00%      4716         -
# 3  Short offline       Completed without error       00%      1896         -
# 4  Extended offline    Completed without error       00%        88         -
# 5  Short offline       Completed without error       00%        15         -
# 6  Short offline       Aborted by host               90%        12         -
# 7  Short offline       Aborted by host               90%        12         -
# 8  Short offline       Aborted by host               90%        12         -
# 9  Short offline       Aborted by host               90%         5         -
#10  Short offline       Aborted by host               90%         5         -
#11  Short offline       Aborted by host               90%         5         -
#12  Short offline       Aborted by host               90%         4         -
#13  Short offline       Aborted by host               90%         4         -
#14  Short offline       Aborted by host               90%         4         -
#15  Short offline       Aborted by host               90%         0         -
#16  Short offline       Aborted by host               90%         0         -

[root@nas ~]# smartctl -l selftest /dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      3118         -
# 2  Short offline       Completed without error       00%        12         -

[root@nas ~]# smartctl -l selftest /dev/sdg
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6298         -
# 2  Extended offline    Completed without error       00%      2555         -

[root@nas ~]# smartctl -l selftest /dev/sdh
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6147         -
# 2  Extended offline    Completed without error       00%      2404         -
# 3  Short offline       Completed without error       00%         0         -

[root@nas ~]# smartctl -l selftest /dev/sdi
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.7.0-1.el7.elrepo.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      6147         -
# 2  Extended offline    Completed without error       00%      2404         -
# 3  Short offline       Completed without error       00%         0         -



[root@nas ~]# smartctl -a /dev/sdb | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdc | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdd | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sde | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1258
[root@nas ~]# smartctl -a /dev/sdf | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdg | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdh | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
[root@nas ~]# smartctl -a /dev/sdi | grep "Raw_Read_Error_Rate"
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0

 

Yup. /dev/sde seems to have an issue.



Get serial number of all drives for a possible RMA on /dev/sde.

[root@nas ~]# smartctl -a /dev/sdb | grep "Serial Number:"
Serial Number:    WD-WMC4N0J0YT1V
[root@nas ~]# smartctl -a /dev/sdc | grep "Serial Number:"
Serial Number:    WD-WMC4N0J2L138
[root@nas ~]# smartctl -a /dev/sdd | grep "Serial Number:"
Serial Number:    WD-WCC4N2FJRTU9
[root@nas ~]# smartctl -a /dev/sde | grep "Serial Number:"
Serial Number:    WD-WCC4N4SSDRFN
[root@nas ~]# smartctl -a /dev/sdf | grep "Serial Number:"
Serial Number:    WD-WCC4N1VYZH52
[root@nas ~]# smartctl -a /dev/sdg | grep "Serial Number:"
Serial Number:    WD-WMC4N0M57KEY
[root@nas ~]# smartctl -a /dev/sdh | grep "Serial Number:"
Serial Number:    WD-WCC4N5YF2Z2Y
[root@nas ~]# smartctl -a /dev/sdi | grep "Serial Number:"
Serial Number:    WD-WCC4N5CJ6H8U




Get List of BadBlocks on all drives. Run in background and save to file.

badblocks -v /dev/sdb > /tmp/bad-blocks-b.txt &
badblocks -v /dev/sdc > /tmp/bad-blocks-c.txt &
badblocks -v /dev/sdd > /tmp/bad-blocks-d.txt &
badblocks -v /dev/sde > /tmp/bad-blocks-e.txt &
badblocks -v /dev/sdf > /tmp/bad-blocks-f.txt &
badblocks -v /dev/sdg > /tmp/bad-blocks-g.txt &
badblocks -v /dev/sdh > /tmp/bad-blocks-h.txt &
badblocks -v /dev/sdi > /tmp/bad-blocks-i.txt &

Monitor file size with:

[root@nas ~]# watch ls -lsa /tmp/bad-blocks-*.txt

If you have a really bad drive it could create a file the size of the drive itself so be sure to monitor and make sure you do not fill up your /tmp directory.

If you need to kill it then ket the pid with:

[root@nas tmp]# ps -ef | grep "badblocks"
UID        PID  PPID  C STIME TTY      TIME     CMD
root     27013 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdb
root     27014 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdc
root     27015 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdd
root     27016 25404  2 10:43 pts/0    00:01:11 badblocks -v /dev/sde
root     27017 25404  3 10:43 pts/0    00:01:13 badblocks -v /dev/sdf
root     27018 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdg
root     27019 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdh
root     27020 25404  3 10:43 pts/0    00:01:12 badblocks -v /dev/sdi
root     31044 26976  0 11:22 pts/1    00:00:00 grep --color=auto badblocks



While badblock test is running I have already got a RMA number from WD and a shipping label on my printer. I ordered a new drive from Amazon that will be here on the 28th. I will swap out then and ship bad drive back on the 29th.

Running smartctl test long on all drives

smartctl -t long /dev/sdb
smartctl -t long /dev/sdc
smartctl -t long /dev/sdd
smartctl -t long /dev/sde
smartctl -t long /dev/sdf
smartctl -t long /dev/sdg
smartctl -t long /dev/sdh
smartctl -t long /dev/sdi

Check progress of test

smartctl -a /dev/sdb | grep "Self-test execution status"
smartctl -a /dev/sdb | grep "of test remaining."

smartctl -a /dev/sdc | grep "Self-test execution status"
smartctl -a /dev/sdc | grep "of test remaining."

smartctl -a /dev/sdd | grep "Self-test execution status"
smartctl -a /dev/sdd | grep "of test remaining."

smartctl -a /dev/sde | grep "Self-test execution status"
smartctl -a /dev/sde | grep "of test remaining."

smartctl -a /dev/sdf | grep "Self-test execution status"
smartctl -a /dev/sdf | grep "of test remaining."

smartctl -a /dev/sdg | grep "Self-test execution status"
smartctl -a /dev/sdg | grep "of test remaining."

smartctl -a /dev/sdh | grep "Self-test execution status"
smartctl -a /dev/sdh | grep "of test remaining."

smartct
l -a /dev/sdi | grep "Self-test execution status"
smartctl -a /dev/sdi | grep "of test remaining."



New drive is in and I am backing up NAS. My btrfs pool of /myraid is gettin backuped up to a PC with raid1.
I am also duplicating the more inportant files to a SSD. Better safe than sorry.

New drive has had a full surface test (9 hours) and passed with flying colors.

I am not turning this into a drive remove/replace since i want to change the partions of my boot SSD so I am just
goging to nuke the entire system and rebuild from scratch.

my /(root) partion is getting a backup vi tar so i can have access to my old crontab files and maintemce scripts.
I can just yse the old samba.conf as well, etc....



tar -zcvpf /myraid/nas.backup.tar.gz --exclude=/myraid --exclude=/usr --exclude=/proc --exclude=/lib --exclude=/lib64 --exclude=/dev /

Now copy the tar.gz file to a few drives off the server as well.

Make USB install Centos 7.3 min and do the install :)

Install done. I see all 9 drives and 4 network connections. I setup all the NICs during the install and they all seem to be ok
Possibily some tweeking on these later.

I installed the OS on my SSD with 1GB /boot and /boot/efi (I am using EFI). The rest to /

My other 8 drives are on my Supermicro AOC-SAS2LP-MV8 JBOD HBA. I will not touch those untill I get ready to setup btrfs on them.

So now some base stuff


cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux

cp /etc/selinux/config /etc/selinux/config.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

systemctl disable firewalld
systemctl stop firewalld

service iptables stop
service ip6tables stop
chkconfig iptables off
chkconfig ip6tables off

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git vim gdisk parted

yum -y groupinstall "Development Tools"
yum -y update
yum -y upgrade

cd /root
echo ':color desert' > .vimrc

systemctl disable kdump.service

reboot

# cat /etc/default/grub
GRUB_TIMEOUT=60
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl_bcache/root ipv6.disable=1 zswap.enable=1 consoleblank=0"
GRUB_DISABLE_RECOVERY="true"

Make changes as refelected about to
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=cl_bcache/root ipv6.disable=1 zswap.enable=1 consoleblank=0"

# grub2-mkconfig -o /boot/grub2/grub.cfg
or if using UEFI
# grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

reboot


Now update kerenel

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
yum install yum-plugin-fastestmirror

yum --enablerepo=elrepo-kernel install kernel-ml

reboot

Manualy select new kernel from grub boot screen.

uname -r
4.10.1-1.el7.elrepo.x86_64

Do any testing you need and when happy set this to default entry when happy.

grub2-set-default 0

reboot

uname -r
4.10.1-1.el7.elrepo.x86_64


Now time to physicaly replace drive then setup btrfs.

poweroff

New drive in and server rebooted

parted -l

shows all drives. 7x WD RED NAS drives show still brtfs partitions. New drive has nothing.

idle3ctl show new drive has head parking on. lets turn that off.

# ./idle3ctl /dev/sde
Idle3 timer set to 138 (0x8a)

# ./idle3ctl -d /dev/sde
Idle3 timer disabled
Please power cycle your drive off and on for the new setting to be taken into account. A reboot will not be enough!

So lets powerr off, let set for a min, and power back on and rechedck.

poweroff

looks good on all drives 8 WD RED drives

[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdb
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdc
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdd
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sde
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdf
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdg
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdh
Idle3 timer is disabled
[root@nas idle3-tools-0.9.1]# ./idle3ctl /dev/sdi
Idle3 timer is disabled


No lets clean the disk for a new array

I used parted to rm the partiions then a w and q.

Then

wipefs -a /dev/sdb
wipefs -a /dev/sdc
wipefs -a /dev/sdd
wipefs -a /dev/sde
wipefs -a /dev/sdf
wipefs -a /dev/sdg
wipefs -a /dev/sdh
wipefs -a /dev/sdi

Then

dd if=/dev/zero of=/dev/sdb bs=1024 count=1024
dd if=/dev/zero of=/dev/sdc bs=1024 count=1024
dd if=/dev/zero of=/dev/sdd bs=1024 count=1024
dd if=/dev/zero of=/dev/sde bs=1024 count=1024
dd if=/dev/zero of=/dev/sdf bs=1024 count=1024
dd if=/dev/zero of=/dev/sdg bs=1024 count=1024
dd if=/dev/zero of=/dev/sdh bs=1024 count=1024
dd if=/dev/zero of=/dev/sdi bs=1024 count=1024

Then to just look at the devs

ls -lsa /dev/sd*
0 brw-rw---- 1 root disk 8,   0 Mar  1 15:02 /dev/sda
0 brw-rw---- 1 root disk 8,   1 Mar  1 15:02 /dev/sda1
0 brw-rw---- 1 root disk 8,   2 Mar  1 15:02 /dev/sda2
0 brw-rw---- 1 root disk 8,   3 Mar  1 15:02 /dev/sda3
0 brw-rw---- 1 root disk 8,  16 Mar  1 15:11 /dev/sdb
0 brw-rw---- 1 root disk 8,  32 Mar  1 15:11 /dev/sdc
0 brw-rw---- 1 root disk 8,  48 Mar  1 15:11 /dev/sdd
0 brw-rw---- 1 root disk 8,  64 Mar  1 15:11 /dev/sde
0 brw-rw---- 1 root disk 8,  80 Mar  1 15:11 /dev/sdf
0 brw-rw---- 1 root disk 8,  96 Mar  1 15:11 /dev/sdg
0 brw-rw---- 1 root disk 8, 112 Mar  1 15:11 /dev/sdh
0 brw-rw---- 1 root disk 8, 128 Mar  1 15:11 /dev/sdi



fdisk -l also shows they look ready


[root@nas idle3-tools-0.9.1]# fdisk -l
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk /dev/sda: 128.0 GB, 128035676160 bytes, 250069680 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048      2099199      1G  EFI System      EFI System Partition
 2      2099200      4196351      1G  Microsoft basic
 3      4196352    250068991  117.2G  Linux LVM      

Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdf: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdg: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdh: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdi: 3000.6 GB, 3000592982016 bytes, 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/cl_nas-root: 125.9 GB, 125883645952 bytes, 245866496 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes



Check out btrfs version first

# btrfs --version
btrfs-progs v4.4.1

I think there is a newer on out. Lets go see.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

cd btrfs-progs

yum -y install libuuid-devel libattr-devel zlib-devel libacl-devel e2fsprogs-devel libblkid-devel lzo* asciidoc xmlto

./autogen.sh
./configure
make

Lets check version from within the folder

[root@nas btrfs-progs]# ./btrfs --version
btrfs-progs v4.9.1

Yup its newer

Now check from /

[root@nas btrfs-progs]# cd /
[root@nas /]# btrfs --version
btrfs-progs v4.4.1
[root@nas /]#

So we got two versions

I copied all +x files from /root/btrfs-progs to /usr/sbin overright files if they exist.

Now from / of drive I get

[root@nas /]# btrfs --version
btrfs-progs v4.9.1

I hope thats good :)

So lets build an array!!!

First I will use raid0 for some quick testing.

mkfs.btrfs -f -m raid0 -d raid0 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

also here is for raid10

mkfs.btrfs -f -m raid10 -d raid10 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

[root@nas ~]# mkfs.btrfs -f -m raid0 -d raid0 -L myraid /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi
btrfs-progs v4.9.1
See http://btrfs.wiki.kernel.org for more information.

Label:              myraid
UUID:               5a5610aa-2615-4ee2-bd4a-076ab2931b70
Node size:          16384
Sector size:        4096
Filesystem size:    21.83TiB
Block group profiles:
  Data:             RAID0             8.00GiB
  Metadata:         RAID0             4.00GiB
  System:           RAID0            16.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  8
Devices:
   ID        SIZE  PATH
    1     2.73TiB  /dev/sdb
    2     2.73TiB  /dev/sdc
    3     2.73TiB  /dev/sdd
    4     2.73TiB  /dev/sde
    5     2.73TiB  /dev/sdf
    6     2.73TiB  /dev/sdg
    7     2.73TiB  /dev/sdh
    8     2.73TiB  /dev/sdi
   
[root@nas ~]# btrfs fi show
Label: 'myraid'  uuid: 5a5610aa-2615-4ee2-bd4a-076ab2931b70
        Total devices 8 FS bytes used 112.00KiB
        devid    1 size 2.73TiB used 1.50GiB path /dev/sdb
        devid    2 size 2.73TiB used 1.50GiB path /dev/sdc
        devid    3 size 2.73TiB used 1.50GiB path /dev/sdd
        devid    4 size 2.73TiB used 1.50GiB path /dev/sde
        devid    5 size 2.73TiB used 1.50GiB path /dev/sdf
        devid    6 size 2.73TiB used 1.50GiB path /dev/sdg
        devid    7 size 2.73TiB used 1.50GiB path /dev/sdh
        devid    8 size 2.73TiB used 1.50GiB path /dev/sdi

Lets mount this thing

mkdir /myraid

mount with UUID from above. uuid: 5a5610aa-2615-4ee2-bd4a-076ab2931b70

mount -t btrfs -o defaults,nodatacow,noatime,x-systemd.device-timeout=30 -U 5a5610aa-2615-4ee2-bd4a-076ab2931b70 /myraid

[root@nas ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                  32G     0   32G   0% /dev
tmpfs                     32G     0   32G   0% /dev/shm
tmpfs                     32G  8.9M   32G   1% /run
tmpfs                     32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/cl_nas-root  118G  2.2G  116G   2% /
/dev/sda2               1014M  191M  824M  19% /boot
/dev/sda1               1022M  9.5M 1013M   1% /boot/efi
tmpfs                    6.3G     0  6.3G   0% /run/user/0
/dev/sdb                  22T   20M   22T   1% /myraid

Oh ya!!! 22TB of btrfs array

My line for fstab i will put in later is:

UUID=5a5610aa-2615-4ee2-bd4a-076ab2931b70   /myraid   btrfs  defaults,nodatacow,noatime,x-systemd.device-timeout=30  0 0


this is how to clear cache when testing transfer speeds to make sure you are not using cache.
Do this between each transfer


sync; echo 2 > /proc/sys/vm/drop_caches
sync; echo 3 > /proc/sys/vm/drop_caches

---tune 10Gb CNA if needed

service irqbalance stop
service cpuspeed stop
chkconfig irqbalance off
chkconfig cpuspeed off
systemctl disable irqbalance
systemctl disable cpuspeed
systemctl stop irqbalance
systemctl stop cpuspeed

vi /etc/sysconfig/network-scripts/ifcfg-eth???
MTU="9000"

vi /etc/sysctl.conf
# -- tuning -- #
# Increase system file descriptor limit
fs.file-max = 65535

# Increase system IP port range to allow for more concurrent connections
net.ipv4.ip_local_port_range = 1024 65000

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

reboot and test speed.

on linux client pointing to server with ip 192.168.90.100

# iperf3 -c 192.168.90.100 -p 5201

on linux server with IP 192.168.90.100

iperf3 -s -p 5201 -B 192.168.90.100

---end tune 10Gb CNA if needed


---setup NFS for ESXi server

vi /etc/exports
/myraid/     192.168.10.0/24(rw,async,no_root_squash,no_subtree_check)
/myraid/     192.168.90.0/24(rw,async,no_root_squash,no_subtree_check)

systemctl start rpcbind nfs-server
systemctl enable rpcbind nfs-server

---end setup NFS for ESXi server




--install samaba if needed

yum -y install samba

useradd samba -s /sbin/nologin

smbpasswd -a samba
            Supply a password
            Retype the password
   
mkdir /myraid

chown -R samba:root /myraid/

vi /etc/samba/smb.conf

[global]
workgroup = WORKGROUP ;use name of your workgroup here
server string = Samba Server Version %v
netbios name = NAS

Add this to botton of /etc/samba/smb.conf file

[NAS]
comment = NAS
path = /myraid
writable = yes
valid users = samba


systemctl start smb
systemctl enable smb
systemctl start nmb
systemctl enable nmb

testparm
   
--end install samaba if needed



---install plex if needed


visit plex site and get rpm for your version of OS
copy this to /root

yum -y localinstall name.rpm

systemctl enable plexmediaserver
systemctl start plexmediaserver

---end install plex if needed



---install LAMP

yum -y install httpd mariadb-server mariadb php php-mysql
systemctl enable httpd.service
systemctl start httpd.service
systemctl status httpd.service

Make sure it works with:
http://your_server_IP_address/

systemctl enable mariadb
systemctl start mariadb
systemctl status mariadb
mysql_secure_installation

vi /var/www/html/info.php
<?php phpinfo(); ?>

http://your_server_IP_address/info.php


---End install LAMP


---Extra goodies

yum -y install epel-release
yum -y install stress htop iftop iotop hddtemp smartmontools iperf3 sysstat mlocate
yum -y update


updatedb **this is to update mlocate db


---End Extra goodies




---Use gmail as relay for sending mail

Replace glen@gmail.com with a real email address in items below
Replace mycentserver.mydomain.domain with real hostname in items below
Replace gmail_password with real password in items below

# yum remove postfix

Now install ssmtp.

# yum -y install ssmtp mailx

Now edit your  /etc/ssmtp/ssmtp.conf. I removed everything and just added the below in the file.

#  vi /etc/ssmtp/ssmtp.conf

root=glen@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=gmail.com
hostname=mycentserver.mydomain.domain
UseTLS=Yes
UseSTARTTLS=Yes
AuthUser=glen@gmail.com
AuthPass=gmail_password
FromLineOverride=YES

# This solved if you get a ssmtp: Cannot open smtp.gmail.com:587 when try to send an email
# if you enabled uncommenting DEBUG=Yes line and your /var/log/maillog show
# SSL not working: certificate verify failed (20) Uncomment the following line but first
# VERIFY FILE EXISTS
TLS_CA_File=/etc/pki/tls/certs/ca-bundle.crt

# DEBUG=Yes

Now edit your /etc/ssmtp/revaliases file and add the following.

# vi /etc/ssmtp/revaliases

root:glen@gmail.com:smtp.gmail.com:587

Now run

# alternatives --config mta

And choose the number for sendmail.ssmtp, like below

There is 1 program that provides 'mta'.

  Selection    Command
-----------------------------------------------
*+ 1           /usr/sbin/sendmail.ssmtp

Enter to keep the current selection[+], or type selection number: 1
#

Now send email to your gmail account from Centos cli

# mail -s "Test Subject" glen@gmail.com

Type your message text and on new line press ctrl d to send

---End Use gmail as relay for sending mail


Thursday, March 2, 2017

Centos 7 backing up my NAS

I put the tar in /myraid (my btrfs 8 drive raid10) then copy this off to another machine that then replicates to cloud.

So if my boot drive goes out I have the backup on the array, on another onsite machine, and in the cloud.

tar --exclude='/myraid' --exclude='/proc' --exclude='/sys' -czvf /myraid/nas_backup.tar.gz /

I don't have enough cloud storage to replicate /myraid but I copy the entire /myraid mount to another onsite machine with a raid1. I guess its better than nothing :)

Saturday, February 25, 2017

Centos 7 with two different networks

Centos 7 with two different networks

I have a vm image with two nics in it. One to my 10Gb 192.168.70.0 network. This network is not on a switch but simply a DAC to my PC. So my PC is 192.168.70.5 and my ESXi is 192.168.70.90.

I also have another network in the vm inage on my 1Gb 192.168.10.0 network. This is my may main network, with a switch, and a route out to internet (192.168.10.1). So my PC is 192.168.10.5, ESXi 192.168.10.90, and router 192.168.10.1.

IP addresses for the vm image itself are:
192.168.10.22
192.168.70.22


Problem is when I have the 192.168.70.22 enabled I am not able to reach the internet. To remedy this I have the folowing /etc/sysconfig/network-scripts ifcfg files:

# cat ifcfg-ens192
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=dcc2277d-a060-48a0-8829-350abf95e269
DEVICE=ens192
ONBOOT=yes
DNS1=192.168.10.1
IPADDR=192.168.70.22
PREFIX=24
#GATEWAY=192.168.70.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no


# cat ifcfg-ens224
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens224
UUID=78e2b2a8-d444-4fd2-9751-bb108861392a
DEVICE=ens224
ONBOOT=yes
IPADDR=192.168.10.22
PREFIX=24
GATEWAY=192.168.10.1
DEFROUTE=yes
DNS1=192.168.10.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_PRIVACY=no


I then change the value of rp_filter to loose mode, value 2 with:

# sysctl net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.all.rp_filter = 2

Restart network with:

# systemctl restart network

I can now ping internet (8.8.8.8), all nodes on 192.168.10.0, and all nodes on 192.168.70.0 (ESXi 192.168.70.90, PC 192.168.70.5 and the vm image of 192.168.70.22).

A quick reboot and re test of access and I am good.

I also want to tune the 10Gb interface as Im wanting to to test SSD disk caching on this image and need the fastest I can get. FYI My PC on 192.168.70.5 has m.2 512GB 950 Pro. I am using a 2.5" 512GB 850 Pro for cache on the vm image of 192.168.70.22. This 850 Pro is in the ESXi server.

So now tune the 10Gb image:

---turn off NetworkManage     

systemctl stop NetworkManager
systemctl disable NetworkManager
systemctl stop NetworkManager-wait-online
systemctl disable NetworkManager-wait-online
systemctl enable network
chkconfig network on
systemctl start network


reboot and sanity check

systemctl status NetworkManager
systemctl status network

---end turn off NetworkManager

---tune 10Gb CNA

service irqbalance stop
service cpuspeed stop
chkconfig irqbalance off
chkconfig cpuspeed off
systemctl disable irqbalance
systemctl disable cpuspeed
systemctl stop irqbalance
systemctl stop cpuspeed

vi /etc/sysconfig/network-scripts/ifcfg-eth???
MTU="9000"

vi /etc/sysctl.conf
# -- tuning -- #
# Increase system file descriptor limit
fs.file-max = 65535

# Increase system IP port range to allow for more concurrent connections
net.ipv4.ip_local_port_range = 1024 65000

# -- 10gbe tuning from Intel ixgb driver README -- #

# turn off selective ACK and timestamps
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0

# memory allocation min/pressure/max.
# read buffer, write buffer, and buffer space
net.ipv4.tcp_rmem = 10000000 10000000 10000000
net.ipv4.tcp_wmem = 10000000 10000000 10000000
net.ipv4.tcp_mem = 10000000 10000000 10000000

net.core.rmem_max = 524287
net.core.wmem_max = 524287
net.core.rmem_default = 524287
net.core.wmem_default = 524287
net.core.optmem_max = 524287
net.core.netdev_max_backlog = 300000

---end tune 10Gb CNA 

Thursday, February 23, 2017

Asterisk 14 on Centos 7.3

Asterisk 14 on Centos 7.3

After fresh of install of Centos 7.3 minimal.

cd /root

cp /etc/sysconfig/selinux /etc/sysconfig/selinux.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/sysconfig/selinux

cp /etc/selinux/config /etc/selinux/config.bak
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

systemctl disable firewalld
systemctl stop firewalld

service iptables stop
service ip6tables stop
chkconfig iptables off
chkconfig ip6tables off

yum -y install bind-utils traceroute net-tools ntp* gcc glibc glibc-common gd gd-devel make net-snmp openssl-devel xinetd unzip libtool* make patch perl bison flex-devel gcc-c++ ncurses-devel flex libtermcap-devel autoconf* automake* autoconf libxml2-devel cmake sqlite* wget ntp* lm_sensors ncurses-devel qt-devel hmaccalc zlib-devel binutils-devel elfutils-libelf-devel wget bc gzip uuid* libuuid-devel jansson* libxml2* sqlite* openssl* lsof NetworkManager-tui mlocate yum-utils kernel-devel nfs-utils tcpdump git vim

yum -y groupinstall "Development Tools"
yum -y update
yum -y upgrade

cd /root
echo ':color desert' > .vimrc


reboot

cd /root

yum -y install epel-release
yum -y install stress htop iftop iotop hddtemp smartmontools iperf3 sysstat mlocate
updatedb

yum -y install gcc gcc-c++ php-xml php php-mysql php-pear php-mbstring mariadb-devel mariadb-server mariadb sqlite-devel lynx bison gmime-devel psmisc tftp-server httpd make ncurses-devel libtermcap-devel sendmail sendmail-cf caching-nameserver sox newt-devel libxml2-devel libtiff-devel audiofile-devel gtk2-devel uuid-devel libtool libuuid-devel subversion kernel-devel kernel-devel-$(uname -r) git subversion kernel-devel php-process crontabs cronie cronie-anacron wget vim

wget http://www.digip.org/jansson/releases/jansson-2.7.tar.gz

tar -xzvf jansson-2.7.tar.gz

cd jansson-2.7

./configure -prefix=/usr

make clean

make && make install

ldconfig

cd /root

wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-14-current.tar.gz

tar xvf asterisk-14-current.tar.gz

cd asterisk-14.3.0

./configure -libdir=/usr/lib64

make menuselect

contrib/scripts/get_mp3_source.sh

make

make install

make progdocs

make samples

make config

systemctl restart asterisk

systemctl status asterisk

asterisk -r

Sunday, February 19, 2017

Broadsoft useful commands

Broadsoft useful commands

Backup servers:

bwadmin@ns1.whittenberg.domain$ locate bwBackup.pl

/usr/local/broadworks/NS_Rel_20.sp1_1.606/bin/bwBackup.pl

bwadmin@ns1.whittenberg.domain$ /usr/local/broadworks/NS_Rel_20.sp1_1.606/bin/bwBackup.pl NetworkServer /tmp/NetworkServer.bak

via the cli is:
NS_CLI/Maintenance/Tools> backupdb /tmp/NetworkServer.bak

Also see
bwRestore.pl
NS_CLI/Maintenance/Tools> restoredb

owncloud helpful commands

Add a trusted domain.:
http://192.168.80.100/owncloud/index.php/settings/admin?trustDomain=192.168.10.100

Clean trashbin/deleted files:
sudo -u apache /var/www/html/owncloud/occ trashbin:cleanup
sudo -u apache /var/www/html/owncloud/occ trashbin:cleanup user1 user2

Take out of maintenance mode:

vim /var/www/html/owncloud/config/config.php

<?php
$CONFIG = array (
  'updatechecker' => false,
  'instanceid' => 'oce24qfiihel',
  'passwordsalt' => 'Dy+BHYATWwB+F4MYYBjPjYX44iXB3o',
  'secret' => 'Bpf9UEgisK4XJecVxrL76ZAwg4n9NnG6RXWO2B4MKzpJNGMz',
  'trusted_domains' =>
  array (
    0 => '192.168.80.100',
    1 => '192.168.10.100',
  ),
  'datadirectory' => '/myraid/owncloud/',
  'overwrite.cli.url' => 'http://192.168.80.100/owncloud',
  'dbtype' => 'sqlite3',
  'version' => '9.1.4.2',
  'logtimezone' => 'UTC',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

Friday, February 17, 2017

owncloud take out of maintenance mode

owncloud take out of maintenance mode

vim /var/www/html/owncloud/config/config.php

<?php
$CONFIG = array (
  'updatechecker' => false,
  'instanceid' => 'oce24qfiihel',
  'passwordsalt' => 'Dy+BHYATWwB+F4MYYBjPjYX44iXB3o',
  'secret' => 'Bpf9UEgisK4XJecVxrL76ZAwg4n9NnG6RXWO2B4MKzpJNGMz',
  'trusted_domains' =>
  array (
    0 => '192.168.80.100',
    1 => '192.168.10.100',
  ),
  'datadirectory' => '/myraid/owncloud/',
  'overwrite.cli.url' => 'http://192.168.80.100/owncloud',
  'dbtype' => 'sqlite3',
  'version' => '9.1.4.2',
  'logtimezone' => 'UTC',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

Thursday, February 9, 2017

Install Linux Kernel 4.9 In CentOS

Install Linux Kernel 4.9 In CentOS

Ripped from: https://www.ostechnix.com/install-linux-kernel-4-9-centos-ubuntu/ 

 

To install the latest kernel, we need to add ELRepo repository.

First,  let us add ELRepo GPG key:
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Then, add ELRepo in CentOS 7 / RHEL 7 / Scientific Linux 7 using command:
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm

To Install ELRepo for CentOS 6 / RHEL 6 / Scientific Linux 6:
rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
To Install ELRepo for CentOS 5 / RHEL 5 / Scientific Linux 6:
rpm -Uvh http://www.elrepo.org/elrepo-release-5-5.el5.elrepo.noarch.rpm

Enable ELRepo fastest mirror by installing the following package:
yum install yum-plugin-fastestmirror

Sample output:
 
[...]
Dependencies Resolved

================================================================================
 Package Arch Version Repository
 Size
================================================================================
Updating:
 yum-plugin-fastestmirror noarch 1.1.31-40.el7 base 32 k

Transaction Summary
================================================================================
Upgrade 1 Package

Total download size: 32 k
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for base
yum-plugin-fastestmirror-1.1.31-40.el7.noarch.rpm | 32 kB 00:02 
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
 Updating : yum-plugin-fastestmirror-1.1.31-40.el7.noarch 1/2 
 Cleanup : yum-plugin-fastestmirror-1.1.31-34.el7.noarch 2/2 
 Verifying : yum-plugin-fastestmirror-1.1.31-40.el7.noarch 1/2 
 Verifying : yum-plugin-fastestmirror-1.1.31-34.el7.noarch 2/2 

Updated:
 yum-plugin-fastestmirror.noarch 0:1.1.31-40.el7 

Complete!

We have added the ELRepo. Now, it is time to install Linux kernel 4.9.

Let us find out the current version details with command:
 
uname -r

Sample output:
3.10.0-327.22.2.el7.x86_64

So, the currently installed Linux Kernel version is 3.10.0-327.22.2.el7.x86_64.

Now, let us install Linux Kernel 4.9 by running the following command from Terminal:
yum --enablerepo=elrepo-kernel install kernel-ml


After installing the Kernel, Reboot your system and select the latest Kernel from the Grub boot menu.


Once you successfully logged-in to the system, run the following command to verify whether latest the latest Linux Kernel has been installed or not:
 
uname -r

Sample output:
4.9.0-1.el7.elrepo.x86_64


Congratulations! You have successfully updated the Linux Kernel. Now, your system is powered by most recent stable Linux Kernel 4.9 version.

If have encountered with any problems after installing the new Linux Kernel? No worries, reboot your system. Select your previously working Kernel from the Boot menu.


Finally, remove the newly installed Kernel using command:
 
yum remove kernel-ml

That’s it.

Tuesday, January 17, 2017

ESXi 6 install stuck on “Relocating modules and starting up the kernel…




ESXi 6 install stuck on “Relocating modules and starting up the kernel…”


Admittedly, I do not do much VMWare these days, but on occasion I have clients that need to use it. Recently I was installing ESXi 6  on my personal lab to do some testing, when the installation got stuck, the last message being  “Relocating modules and starting up the kernel”

image

So, what’s happening? Granted this is an older Sun 4170M2 server, but  I have installed Oracle VM on the server with no issues. I also have run KVM hypervisors, Windows Hyper-V and Citrix without any issues. Well after a little troubleshooting it looks like VMWare does not like the video subsystem and is getting confused when it is checking to see if the server is headless.

The fix is relatively easy, we have to add a boot parameter to the ESXi kernel. To do this, reboot, and while ESXi is booting hit the tab key

image

Next, we will add a parameter to disable the headless check “ignoreHeadless=TRUE”

image

And now reboot and install normally…

image

Continue the installation normally, but when you get to the end pay attention.

image

Now reboot, and get ready to hit “Shift O” this time

image

The add in the ignoreHeadless=TRUE in the boot parameter list, it should look like this

image

And then hit enter to continue the boot process.
Next, once we are back up we will need to make the ignoreHeadless=TRUE permanent for each reboot.

Log into the ESXi system as root, from the login screen hit F2

image

Login as root

image

Arrow down to troubleshooting options to enable the shell

image

Next enable the ESXi shell

image

The enter key will toggle the shell state.

image

Now from the console hit ALT-F2. On ILOMs you will need to use enable the ALT key from the keyboard menu

image

Then hit F2, to get to the shell login.
Don’t forget to uncheck the alt key from the keyboard dropdown once you are on the shell screen, otherwise you will have issues typing in the password!

image

next login as root, and we will run a single command
esxcfg-advcfg --set-kernel "TRUE" ignoreHeadless

image

I am a big fan of the concept of “trust, but verify”, so let’s verify that the parameter is correct by getting the parameter with the command
“esxcfg-advcfg –get-kernel ignoreHeadless”

image

Now we  are done. type exit to logout, and then ALT-F2 to go back to the main screen

Hopefully this helps you with getting ESX installed on some older systems.