Posts

Security Group Inbound rules list using aws cli

  for region in `aws ec2 describe-regions --query "Regions[].{Name:RegionName}"   --output text` do     :     if [ -z "$(aws ec2 describe-instances --region $region --filter "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId, Tags[?Key=='Name'].Value|[0]]"  --output text | awk '{print $1}')" ]; then         echo "There are no EC2 Instances in the region :: "$region     else         echo "++++++++++++++++++++++++++++++++++++++++++++++++++++"         echo "Checking Amazon Instances on the region "$region         echo "++++++++++++++++++++++++++++++++++++++++++++++++++++"     fi     for instance in `aws ec2 describe-instances --region $region --filter "Name=instance-state-name,Values=running" --query "Reservations[*].Instances[*].[InstanceId, Tags[?Key=='Name'].Value|[0]]"  --output text | awk '{print $1}'`     do     

AWS instance id, ami id and creation date using shell script with array usage and for loop.

 region_name=("us-east-1" "eu-north-1" "ap-south-1"  "eu-west-3" "eu-west-2" "eu-west-1" "ap-northeast-3" "ap-northeast-2" "ap-northeast-1" "sa-east-1" "ca-central-1" "ap-southeast-1" "ap-southeast-2" "eu-central-1" "us-east-1" "us-east-2" "us-west-1" "us-west-2") for region in "${region_name[@]}"  do   : array_us_east_1=($(aws ec2 describe-instances --filters Name=instance-state-name,Values=running,stopped --query "Reservations[*].Instances[*].InstanceId" --output text --region $region))   for instance_id in "${array_us_east_1[@]}"  do    :    echo "InstanceId :" $instance_id    ami_id_us_east_1=($(aws ec2 describe-instances --instance-id  $instance_id --region $region  --query Reservations[].Instances[].[ImageId] --output text))     for ami_id in "${ami_id_us_eas

How to extract java stack trace log from tomcat log file?

catalina.out - Tomcat output file. ERROR - The keyword displayed in the log file containing the java stack trace. sed -n '/ERROR/p; //,/^[0-9]\{4\}-/ {/^[0-9]\{4\}-/!p}' catalina.out  Ref : https://www.unix.com/shell-programming-and-scripting/277433-using-grep-finding-stacktraces.html

How to find java heap memory usage ?

jmap -heap 26442 Attaching to process ID 26442, please wait... Debugger attached successfully. Server compiler detected. JVM version is 8.1.045 9.0.4+011 using thread-local object allocation. Garbage-First (G1) GC with 8 thread(s) Heap Configuration:    MinHeapFreeRatio         = 40    MaxHeapFreeRatio         = 70    MaxHeapSize              = 27262976000 (26000.0MB)    NewSize                  = 5452592 (5.1999969482421875MB)    MaxNewSize               = 16357785600 (15600.0MB)    OldSize                  = 5452592 (5.1999969482421875MB)    NewRatio                 = 2    SurvivorRatio            = 8    MetaspaceSize            = 21807104 (20.796875MB)    CompressedClassSpaceSize = 268435456 (256.0MB)    MaxMetaspaceSize         = 10737418240 (10240.0MB)    G1HeapRegionSize         = 8388608 (8.0MB) Heap Usage: G1 Heap:    regions  = 3250    capacity = 27262976000 (26000.0MB)    used     = 25738931416 (24546.557823181152MB)    free     = 1524044584 (1453.4421768188477MB)    94.40

SSL domain expiry - bulk domain list

Hi, This is where you need to check the SSL certificate validity for large number of domains. domainlist.txt - The file which stores the list of domains that we need to check the SSL expiry. In this file, make sure that only one domain name is in each row. i - The variable used here, for each iteration, each domain is checked. for i in `cat domainlist.txt` ; do echo "++++++++++++++++ $i ++++++++++++++++" ; echo | openssl s_client -servername NAME -connect $i:443 2>/dev/null | openssl x509 -noout -dates ; echo --------------------------------------- ; done

Bash basics for personal use

Replace using sed when too many slashes are there. --------------------------------------------------- sed -i -e 's@/home/username/jas/@/home/username/jas/folder/@g' file.sh sed -i -e 's@xxx.xxx.xxx.xxx@xxx.xxx.xxx.xxx@g' file.sh Note : In bash scripting on remote server ssh login, we could experience problems with awk print $ and "`". In such situations, use cut and there will be no such issues. To add double quotes in a line which has words separated by comma. ----------------------------------------------------------------- sed -e 's/"//g' -e 's/[^,]*/"&"/g' -i filename To add double quotes at end of line in a file. ------------------------------------------------- sed -e 's/$/"/' -i filename To add double quotes at beginning of a line in a file. ----------------------------------------------------- sed -e 's/^/"/' -i filename To check a log file between two dates. -------------------------------

Add hosts entry using python

#Task : To add hosts entry to a server hosts file using python script. # To whomsoever wondering why, this is a task which can be easily achieved via shell script, but I went forward with Python to learn python. #Script to change the remote IP across the hosts file. #How to run this script : python scriptname IP_that_we_need_to_set_hosts_entry #Importing system library for passing arguments during runtime to incorporate as a jenkins job. import sys #The function add_remote add the remote entry to the hosts file and shows us the new entry that we added. def add_remote():  with open("/etc/hosts", "a") as myfile:   myfile.write( str(entered_IP) + ' remote_server_hostname' + '\n' )  with open("/etc/hosts", "r") as myfile:   for line in myfile.readlines():    if 'remote_server_hostname' in line:     print(line)              #The function fun checks if the hosts entry for remote is added in the server. [Main function] def fu

Shell script to add a public key to server and provide sudo privillege for that user.

#!/bin/bash #function which contains the ssh public key for user admin to add in the servers and another function to check if key already exists #If you want to add your user key, replace the admin with your username and add your ssh key to the section shown below. ################################################################## admin_keyexist_check() { if grep -q "******ssh key goes here******admin@<serverhostname>" /home/admin/.ssh/authorized_keys; then     echo " The ssh key already exists and is shown below...!!"        grep -i "admin" /home/admin/.ssh/authorized_keys     echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"     echo "Removing this shell script now...!!"     rm -fv $0     exit else     echo " The ssh key doesn't exists...!! We will add it...!!"     echo "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" fi } ####

EC2 instance listing using python boto3

#Purpose : To list all the AWS EC2 instances in the specified region. #The script assume that you have a working aws cli configured with privilege to access the aws ec2 instances. #!/usr/bin/env python import boto3 import sys import time region_name = raw_input('Enter aws region for which you want to get the details of the EC2 Instance [sample : us-east-1]: ') print (region_name, type(region_name)) time.sleep(5) ec2 = boto3.resource('ec2', globals()['region_name']) for instance in ec2.instances.all():     print instance.id, instance.state, instance.private_ip_address, instance.public_ip_address, instance.tags

AWS Command line Tasks : Create EC2 instance and EC2 instance with 500GB root volume.

To create an EC2 instance from command line. aws ec2 run-instances  --image-id ami-efe09bf8 --count 1 --instance-type m4.4xlarge --key-name mysshkey --security-group-ids sg-126adasd2--subnet-id subnet-148fd971 --block-device-mapping /dev/sda1=:500:false --region us-east-1 --image-id : The AMI ID which we are going to create the instance. --count : The number of instances that we need to create. --instance-type : The type of EC2 instance that we are going to use it. --key-name : The pem that we are going to use to connect to this server. --secrutiy group : The security group this instance is getting used. --subnet-id : The subnet in which we want the ec2 instance. --block device mapping - device:space in gb : to terminate volume [true or false.] To create an EC2 instance with 500GB root volume. aws ec2 run-instances --image-id ami-efe09bf8 --block-device-mappings  '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":500,"DeleteO

AWS command line tasks - Create 500 GB EBS Volume, to attach and detach the volume

To create a volume of 500 GB   aws ec2 --region us-east-1 create-volume --size 500 --availability-zone us-east-1d --volume-type gp2 size : in GB availability zone : This needs to be in the same availability zone as that of the instance that we are going to attach the EBS volume. To attach an EBS volume to the ec2 instance. aws  ec2 attach-volume --volume-id vol-9dfjhsdkfj93 --instance-id i-1234567899oasd  --device /dev/xvdb --region us-east-1 After attaching format the drive and mount it like what we do for a normal hard disk drive. To detach an EBS volume from the ec2 instance. aws ec2 detach-volume --volume-id vol-9dfjhsdkfj93 --region us-east-1

AWS Command line Tasks - Create SSH key pair, security group, Allow connection to a port

We need a working aws cli configured to use the following commands. To create an SSH pem file. aws ec2 create-key-pair --region us-east-1 --key-name mysshkey --output text > mysshkey.pem  region :  The region in which the ssh pem file is getting created. Here we are saving the pem file in our linux machine as the name mysshkey.pem  Modify the permission of the pem file. chmod 400 mysshkey.pem  To create a security group. aws ec2 create-security-group --region us-east-1 --group-name work_project --description "Project_for_work" --vpc-id vpc-12345  group name - It is the name that we are going to give for the security group description - The description that we are going to give for the security group To allow connection to a port aws ec2 --region us-east-1 authorize-security-group-ingress --group-id sg-12345 --protocol tcp --port 22 --cidr 10.100.0.0/16 aws ec2 --region us-east-1 authorize-security-group-ingress --group-id sg-12345 --protocol tcp --p

High process count notification - shell script

#Author : Jaison # Purpose : To send email to the user when a user hit a specific process count list. process_count="$(ps -u root  -L | wc -l)" if [ "$process_count" -gt "10000" ]; then  Email_Body='Test"   echo "$Email_Body" | /usr/bin/mail -s "Server $HOSTNAME Process Count : $process_count IS A LARGE VALUE FOR  USER. CHECK IMMEDIATELY. " <email_account> fi

useradd shell script - bulk read username and password from file.

# Author : Jaison #Purpose : Add user and set their password from the text file all.txt file_name="all.txt" while read user pass do     #useradd ${user} -p ${pass}     useradd ${user}     echo "${pass}" | passwd --stdin ${user}     echo "Adding user "${user}"  with the password  "${pass} done < $file_name

DNS - A simple explanation.

Hi All, Let me try to throw some pebbles regarding the DNS, one of the simplest concepts on the internet, but if explained properly, it can go into a high level of complexity. DNS is something which is used to identify the devices which are connected to the internet.Before jumping into the concept of DNS, I hope everyone who is reading this chapter is aware of IP Adress. If not, IP Address or in short IP is a set of numbers and symbols used to identify machines connected on the internet. So we already have IP so you might be thinking what is the need of DNS in the world?? Well, that is the problem with the humans, we are good with names but not so good with the numbers. To overcome this problem, we are using DNS. Domain Name System in the point of view of a System Administrator is something unavoidable when it comes to hosting the websites on the server. The simple concept of how it works is explained in my propensity as shown below. When a website is accessed on a web b

Cheat sheet for Hardware RAID health check - Megaraid, Adaptec, 3wareraid and HPraid.

Hi Folks, In another post " raid type " I've mentioned how to find if the server is running with Software or Hardware RAID. Here, we can go ahead and check the health of the Hardware RAID array in the server. The first thing we need to detect is the controller in which the raid is configured and you can easily get it from the another post that I mentioned earlier. However, I'm providing it here again for your convenience. 1. Login to the server as the root user. 2. Execute the following command. /sbin/lspci -vv | grep -i raid 3. This will show the raid controller running on the server. The different types of controller I've worked with includes a. Megaraid which uses megacli b. Adaptec which uses arcconf . c. 3ware raid which uses tw_cli d. HPraid which uses hpacucli . How to check the Megaraid array health status? The Megaraid is usually installed in the /opt/MegaRAID and you can execute the following command to verify the health of the arr

Logical volume vmxxxx_img is used by another device - Error on LVM removal

Hi Folks, I've faced error while trying to remove an LVM from the server. The exact LVM error will be "Logical volume vmxxxx_img is used by another device" on executing the lvremove command. Feel free to remove the following steps to remove the LVM from the server. Note: Please remember to replace <id> with your  VMID in the below section. ---------------------------------------------------------- dmsetup ls dmsetup info -c xen-vm<id>_img dmsetup remove xen-vm<id>_img lvremove -f /dev/xen/vm<id>_img ----------------------------------------------------------

File system corrupted on the OpenVZ container.

It happens rarely but when it does, it is hard to handle. The OpenVZ VM's are usually stable in nature, but I've encountered situations where OpenVZ VM was showing file system error and was refusing to start. You can follow the below steps to recover the VM. 1. ~# vzctl stop ctid 2. # ploop mount /vz/private/ctid/root.hdd/ DiskDescriptor.xml 3. now do a df -h and you will get the ploop id for the VM. 4. fdisk -l /dev/ploop12345 5. e2fsck /dev/ploop12345p1 6. ploop umount -d /dev/ploop12345 7. vzctl start ctid

Change IO Scheduler for SSD and HDD server with Software RAID

Hi Guys, It has been a while... I faced a problem with IO on one of the RAID 1 Linux servers and the IO schedulers were the default cfq in the server. It doesn't matter if the drive is in software RAID or not. If you are using SSD drives, then there are no mechanical parts in the HDD, so you need to change the IO schedulers to noop. If the SSD drive is mounted as /hda, then below command will do it. ---------------------------------------------------------- echo noop > /sys/block/hda/queue/scheduler ---------------------------------------------------------- For the HDD drive, the scheduler that is recommended is `deadline` in case of poor performance problems. ---------------------------------------------------------- echo deadline > /sys/block/hdb/queue/scheduler ---------------------------------------------------------- In case of Software RAID, just make sure that the drive you are changing the scheduler is inside the RAID. You can see the drives ins

Check the number of connections using nf_conntrack or using tcpdump

tcpdump -tnn -ieth0 -c 20000 | awk -F ">" '{print $1}' | awk -F " " '{print $2}' | awk -F "." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr | awk ' $1 > 100 ' Note : If the network interface card in use on the server is eth0 cat /proc/net/nf_conntrack > contoutput && cat  contoutput  | awk '{ print $7,$8 }' | sort | uniq -c | sort -rn | head