Wednesday, June 30, 2021
Azure:Devops Stakeholder:Basic Access
Supported access levels
Assign users or groups of users to one of the following access levels:
- Stakeholder: Provides partial access, can be assigned to unlimited users for free. Assign to users with no license or subscriptions who need access to a limited set of features.
- Basic: Provides access to most features. Assign to users with a Visual Studio Professional subscription, an Azure DevOps Server CAL, and to users for whom you're paying for Basic access in an organization.
- Basic + Test Plans: Provides access to all features included in Basic, as well as Azure Test Plans. Assign to users with a Visual Studio Test Professional or MSDN Platforms subscription, and to users for whom you're paying for Basic + Test Plans access in an organization.
- Visual Studio subscription: Assign to users who already have a Visual Studio subscription. The system automatically recognizes the user's subscription—Visual Studio Enterprise, Visual Studio Professional, Visual Studio Test Professional, or MSDN Platform—and enables any other features that are included in their subscription level. If you assign Basic or Stakeholder, they also receive their Visual Studio subscription benefits upon sign-in.
Tuesday, June 29, 2021
Linux: Unlock User
https://www.2daygeek.com/lock-unlock-disable-enable-user-account-linux/
id <username>
passwd --status <username>
passwd -u <username>
usermod --unlock <username>
AWS : Mount EFS on EC2 - Part II - V Important
sudo mkdir -p /efs
sudo chmod -R ugo+rwx /efs
ls -ld /efs
sudo vi /etc/fstab
fs-face524e.efs.us-east-1.amazonaws.com:/ /efs nfs4 rw,intr,hard,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport
sudo mount -a
sudo reboot now
chmod -R 777 /efs is not safer
chmod -R ugo+rwx /efs is safer - Does not override SETUID AND SETGID BITS - Just Appends permissions
https://unix.stackexchange.com/questions/296675/is-chmod-r-ugorwx-safer-than-chmod-r-777
cd / && sudo umount /efs && sudo mv /efs /fileshare && sudo chmod ugo+w /fileshare && cat /etc/fstab
sudo sed -i 's+/efs+/fileshare+g' /etc/fstab
sudo mount -a && sudo reboot now
ls -l /fileshare
df -h
ls -l /fileshare ; df -h
ls -l /fileshare && df -h
umount /efs
sudo mv /efs /fileshare
sudo chmod ugo+w /fileshare
sudo vi /etc/fstab
sudo reboot now
https://www.cyberciti.biz/faq/how-to-use-sed-to-find-and-replace-text-in-files-in-linux-unix-shell/
sudo sed 's/efs/fileshare/g' /etc/fstab //Replace content and its temporary
sudo sed 's+/efs+/fileshare+g' /etc/fstab // Change Delimiter
sudo sed -i 's+/efs+/fileshare+g' /etc/fstab //-i save changes to file/stream
sudo sed -e '/fs-face524e/s/fileshare/helloooo/' /etc/fstab //Replace only if lines contain specific string "fs-face524e"
How to replace a string in file in Linux
Friday, June 25, 2021
Azure Agent - Configure As a Service Linux or Standalone or Nohup &
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops
#Configure it as a linux process and using nohup &
./run.sh
ps -ef --sort=start_time |grep -E 'Agent|run.sh'
--------------------------------------------------------------------------------------------------------
#Configure Listener as a service in Linux, It actually, starts 3 Services
sudo ./svc.sh install #Creates a symlink under /etc/systemctl/system
sudo ./svc.sh start
sudo ./svc.sh status
sudo ./svc.sh uninstall #Uninstall Service(SystemD) ,You should stop before you uninstall.
ps -ef --sort=start_time |grep -E 'Agent|runsvc'
ps -ef --sort=start_time |grep -E 'vsts|Agent|run.sh'
sudo systemctl is-active 'vsts.agent.GenpactDigitalEngineering.eks\x2ddev.dev.service'
sudo systemctl is-enabled 'vsts.agent.GenpactDigitalEngineering.eks\x2ddev.dev.service'
sudo systemctl status 'vsts.agent.GenpactDigitalEngineering.eks\x2ddev.dev.service'
#Configure Listener as a service in Linux, It actually, starts 3 Services
Loaded: loaded (/etc/systemd/system/vsts.agent.GenpactDigitalEngineering.eks\x2ddev.dev.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-06-26 04:12:27 UTC; 17min ago
Main PID: 27333 (runsvc.sh)
Tasks: 21 (limit: 4915)
CGroup: /system.slice/vsts.agent.GenpactDigitalEngineering.eks\x2ddev.dev.service
├─27333 /bin/bash /home/eks/agent/runsvc.sh
├─27336 ./externals/node/bin/node ./bin/AgentService.js
└─27356 /home/eks/agent/bin/Agent.Listener run --startuptype service
Remove and re-configure an agent (Not Service) The Whole Agent
To remove the agent:
1) Stop and uninstall the service as explained above.
2) Remove the agent. [As it appears in Agent Pool]
./config.sh remove [Enter your credentials.]
After you've removed the agent, you can configure it again.
Thursday, June 24, 2021
AWS : S3API vs S3
aws s3api create-bucket --bucket my-bucket --region us-east-1
aws s3 mb s3://myeucentral1bucket --region eu-central-1
aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET
--------------------------------------------------
aws s3api create-bucket --bucket cca-product-dev-s3bucket --region us-east-1 --acl public-read
aws s3 sync s3://cca-product-s3-bucket s3://cca-product-dev-s3bucket --dryrun
--------------------------------------------------
https://docs.aws.amazon.com/cli/latest/reference/s3api/create-bucket.html
https://aws.amazon.com/blogs/developer/leveraging-the-s3-and-s3api-commands/
https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/
https://stackoverflow.com/questions/27932345/downloading-folders-from-aws-s3-cp-or-sync
difference b/w Linux Operators : | || && & > >> ;
https://unix.stackexchange.com/questions/159489/is-there-a-difference-between-and-and
https://unix.stackexchange.com/questions/89386/what-is-symbol-and-in-unix-linux
> redirects output to a file, overwriting the file.
>> redirects output to a file appending the redirected output at the end
;
: commands separated by a;
are executed sequentially. The shell waits for each command to terminate in turn.&&
: command after&&
is executed if, and only if, command before&&
returns an exit status of zero. You can think of it asAND
operator.|
: a pipe. In expressioncommand1 | command2
The standard output of command1 is connected via a pipe to the standard input of command2.
There are more similar control operators, worth to mention:
||
: command after||
is executed if, and only if, command before||
returns a non-zero exit status. You can think of it asOR
operator. Please note, that|
and||
are completely different animals.&
: the shell executes the command terminated by&
in the background, does not wait for the command to finish and immediately returns exit code 0. Once again,&
has nothing to do with&&
.|&
: a shorthand for2>&1 |
i.e. both standard output and standard error of command1 are connected to command2's standard input through the pipe.
Additionally if you use zsh
then you can also start command with &|
or &!
. In this case job is immediately disowned, after startup it does not have a place in the job table.
Linux : Difference between >> and >
> redirects output to a file, overwriting the file.
>> redirects output to a file appending the redirected output at the end
https://unix.stackexchange.com/questions/89386/what-is-symbol-and-in-unix-linux
chmod octet 777 vs -R ugo+rwx ??? Difference
chmod -R 777 /efs is not safer
chmod -R ugo+rwx /efs is safer - Does not override SETUID AND SETGID BITS - Just Appends permissions
https://unix.stackexchange.com/questions/296675/is-chmod-r-ugorwx-safer-than-chmod-r-777
sudo tee -a
Append text when using sudo
echo '104.20.186.5 www.cyberciti.biz' | sudo tee -a /etc/hosts
Persist firewal changes iptables redhat 7 across restart ?
Persist iptables redhat 7 ?
https://www.thegeekdiary.com/centos-rhel-how-to-make-iptable-rules-persist-across-reboots/
[IMPORTANT]
https://serverfault.com/questions/708728/iptables-not-starting-upon-reboot
https://www.tecmint.com/linux-firewall-iptables-interview-questions-and-answers/ [IMP]
https://linuxconfig.org/how-to-change-a-runlevel-on-rhel-7-linux-system
Typically the location of iptables configuration lies at ‘/etc/sysconfig/iptables‘ whereas firewalld configuration lies at ‘/etc/firewalld/‘, which is a set of XML files.
EDIT
/etc/firewalld/zones/public.xml
sudo systemctl status firewalld
sudo /sbin/service iptables status
sudo /sbin/service iptables save
sudo less /etc/sysconfig/iptables
sudo iptables -S
sudo iptables -A IN_public_allow -p tcp -m tcp --dport 27017 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 8080 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
<service name="ssh"/>
<service name="dhcpv6-client"/>
<port protocol="tcp" port="8080"/>
Wednesday, June 23, 2021
Redhat :RPM-Build/Make : How to make RPM File from Git Source Code : Make and RPM-Build [ make rpm]
https://docs.aws.amazon.com/efs/latest/ug/installing-amazon-efs-utils.html#installing-other-distro
sudo yum list installed |grep amazon-efs-utils
sudo yum -y install make
sudo yum -y install rpm-build
cd /path/efs-utils
sudo make rpm
sudo yum -y install ./build/amazon-efs-utils*rpm
AWS Configure : .aws : credentials : config :role_Arn : source_profile: credential_source
https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#the-shared-credentials-file
%UserProfile%/.aws [Windows]
~/.aws [Linux]
All these defaults can be changed too
aws configure [Command to add Secret Key/Access Key using command Line itself]
role_arn and either a source_profile or a credential_source.
- Shared Credentials File
- AWS CLI Conflig File
They both contain Access_Key and Secret_Key
while
The shared credentials file has a default location of ~/.aws/credentials.
You can change the location of the shared credentials file by setting the AWS_SHARED_CREDENTIALS_FILE environment variable.
The AWS CLI config file, which defaults to ~/.aws/config has the following format:
[default]
aws_access_key_id=foo
aws_secret_access_key=bar
region=us-west-2
Profiles that aren't the default profile are specified by creating a section titled "profile profilename":
[profile testing]
aws_access_key_id=foo
aws_secret_access_key=bar
region=us-west-2
aws_access_key_id, aws_secret_access_key, aws_session_token. These are the only supported values in the shared credential file. Also note that the section names are different than the AWS CLI config file (~/.aws/config). In the AWS CLI config file, you create a new profile by creating a section of [profile profile-name], for example:
[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
In the shared credentials file, profiles are not prefixed with profile, for example:
[development]
aws_access_key_id=foo
aws_secret_access_key=bar
Credentials specified in the shared credentials file have precedence over credentials in the AWS CLI config file
Shared Credentials File > Config File
Precedence
The above configuration values have the following precedence:
- Command line options
- Environment variables
- Configuration file
Credentials can be specified in several ways:
- Environment variables
- The AWS Shared Credential File
- The AWS CLI config file
Tuesday, June 22, 2021
AWS : EFS : Mount Target and Mount in EC2
sudo yum -y install nfs-utils
sudo service nfs start
sudo service nfs status
sudo mkdir -p /efs
sudo chmod -R ugo+rwx /efs
ls -ld /efs
echo 'fs-face524e.efs.us-east-1.amazonaws.com:/ /efs nfs4 rw,intr,hard,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0' >> /etc/fstab
sudo mount -a
sudo reboot now
sudo mount -a
#fs-face524e.efs.us-east-1.amazonaws.com:/ /efs nfs4 rw,intr,hard,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0
#echo 'fs-face524e.efs.us-east-1.amazonaws.com:/ /efs nfs4 rw,intr,hard,_netdev,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0' | sudo tee -a /etc/fstab
#sudo vi /etc/fstab
chmod -R 777 /efs is not safer
chmod -R ugo+rwx /efs is safer - Does not override SETUID AND SETGID BITS - Just Appends permissions
https://unix.stackexchange.com/questions/296675/is-chmod-r-ugorwx-safer-than-chmod-r-777
https://www.thegeekdiary.com/understanding-the-configuration-file-for-mounting-file-systems-etc-fstab/
https://askubuntu.com/questions/9939/what-do-the-last-two-fields-in-fstab-mean
The EFS mount helper is part of the amazon-efs-utils package.
The amazon-efs-utils package is an open-source collection of Amazon EFS tools.
For more information, see Manually installing the Amazon EFS client.
https://docs.aws.amazon.com/efs/latest/ug/installing-amazon-efs-utils.html#installing-other-distro
Before the Amazon EFS mount helper was available,
we recommended mounting your Amazon EFS file systems using the standard Linux NFS client.
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-old.html#mounting-fs-install-nfsclient
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-dns-name.html
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-ip-addr.html
https://docs.aws.amazon.com/efs/latest/ug/efs-mount-helper.html#mount-fs-auto-mount-onreboot
To view and copy the exact commands to mount your EFS file system using the mount target IP address
Open the Amazon Elastic File System console at https://console.aws.amazon.com/efs/.
In the Amazon EFS console, choose the file system that you want to mount to display its details page.
To display the mount commands to use for this file system, choose Attach in the upper right.
The Attach screen displays the exact commands to use for mounting the file system.
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html
https://kichik.com/2020/09/08/how-does-ec2-instance-profile-work/
https://computingforgeeks.com/mount-aws-efs-file-system-on-ec2/
https://docs.aws.amazon.com/efs/latest/ug/efs-mount-helper.html#mounting-access-points
sudo mkdir /efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-face524e.efs.us-east-1.amazonaws.com:/ efs
fs-face524e.efs.us-east-1.amazonaws.com:/ efs nfs4 defaults,_netdev 0 0
umount -f efs
sudo mount -a
Mounting on Amazon EC2 with a DNS name
Mounting with an IP address [https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-dns-name.html]
Mounting your Amazon EFS file system automatically[https://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-ip-addr.html]
[https://docs.aws.amazon.com/efs/latest/ug/efs-mount-helper.html#mount-fs-auto-mount-onreboot]
NFS client
nfs-utils for RHEL, CentOS, Amazon Linux, and Fedora distributions
nfs-common for Debian and Ubuntu distributions
Network File Sharing (NFS) is a protocol that allows you to share directories and files with other Linux clients over a network
AWS : EBS Volumes : Attach and Mount on EC2
https://devopscube.com/mount-ebs-volume-ec2-instance/
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
- lsblk
- lsblk -f
- df -h
- less /etc/fstab
[ec2-user@ip-10-79-196-74 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 150G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
└─nvme0n1p2 259:2 0 150G 0 part /
[ec2-user@ip-10-79-196-74 ~]$ lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
nvme0n1
├─nvme0n1p1
└─nvme0n1p2 xfs 77f1de26-38e6-4e1d-8a1e-baa1610669e6 /
[ec2-user@ip-10-79-196-74 ~]$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Mon Oct 28 17:51:10 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=77f1de26-38e6-4e1d-8a1e-baa1610669e6 / xfs defaults 0 0
[ec2-user@ip-10-79-196-74 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 25M 16G 1% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p2 150G 6.5G 144G 5% /
tmpfs 3.1G 0 3.1G 0% /run/user/1000
[ec2-user@ip-10-79-196-74 ~]$
EFS : Mount Instructions are available
Click attach. This opens a page with mount instructions for the EFS.
https://computingforgeeks.com/mount-aws-efs-file-system-on-ec2/
Check if a Package is installed on Linux or not
https://www.cyberciti.biz/faq/apt-get-list-packages-are-installed-on-ubuntu-linux/
https://www.2daygeek.com/find-out-if-package-is-installed-or-not-in-linux-unix/
#RHEL 6, RHEL7
yum list installed openssh
#RHEL 8
dnf list installed httpd
#Ubuntu
apt list -a pkgNameHere
apt list --installed
apt -qq list nano
Amazon Image : Default User
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connection-prereqs.html
Get the default user name for the AMI that you used to launch your instance:
For Amazon Linux 2 or the Amazon Linux AMI, the user name is
ec2-user
.For a CentOS AMI, the user name is
centos
.For a Debian AMI, the user name is
admin
.For a Fedora AMI, the user name is
ec2-user
orfedora
.For a RHEL AMI, the user name is
ec2-user
orroot
.For a SUSE AMI, the user name is
ec2-user
orroot
.For an Ubuntu AMI, the user name is
ubuntu
.Otherwise, if
ec2-user
androot
don't work, check with the AMI provider.
How to make File Immutable in Linux : Even Root cannot delete : CHATTR : LSATTR
[root@ip-10-79-197-70 ec2-user]# sudo chattr +i -V backup/iptables.backup
chattr 1.42.9 (28-Dec-2013)
Flags of backup/iptables.backup set as ----i-----------
[root@ip-10-79-197-70 ec2-user]# sudo lsattr backup
----i----------- backup/iptables.backup
[root@ip-10-79-197-70 ec2-user]# rm -rf backup
rm: cannot remove ‘backup/iptables.backup’: Permission denied
[root@ip-10-79-197-70 ec2-user]# sudo rm -rf backup
rm: cannot remove ‘backup/iptables.backup’: Permission denied
[root@ip-10-79-197-70 ec2-user]# exit
exit
Redhat : IPTABLES : How to persist iptables as its changes are not saved by default
https://www.geeksforgeeks.org/iptables-restore-command-in-linux-with-examples/
7.2.2. SAVING AND RESTORING IPTABLES RULES
/sbin/service iptables save
/etc/sysconfig/iptables
and are applied whenever the service is started or restarted, including when the machine is rebooted.AWS : EFS
https://docs.aws.amazon.com/efs/latest/ug/mounting-fs.html
The EFS mount helper is part of the amazon-efs-utils package. The amazon-efs-utils package is an open-source collection of Amazon EFS tools. For more information, see Manually installing the Amazon EFS client.
Before the Amazon EFS mount helper was available, we recommended mounting your Amazon EFS file systems using the standard Linux NFS client. For more information, see Mounting file systems without the EFS mount helper.
TODO 22 June 2021
- VEA Vault ??
- EFS Mount ?? ************************
- QA -> Approval --- JIRA
- SSL EWS Hema - Nihar ***************************
- Devops Handover CVS --- Atul, Hemang
- Amit Yadav - API ???
- Release Pipeline CAH Mohsin
- Mongodb Service
- Python SDK
- Java Lambda
- How to login to Azure VM ?????
- Ubuntu Script ----> sudo ufw disable
- Instance EFS Linking ??? Mount
Monday, June 21, 2021
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesStopping.html
Cloudwatch -> Metrics -> Explorer
Explorer -> EC2
Explorer -> Elastic Block Store
From -> InstanceId: i-000528b883bf29b49
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-resource-over-utilization/
sudo ufw status
sudo ufw disable
System status checks System reachability check passed
Instance status checks Instance reachability check failed
Note: A stop and start isn't equivalent to a reboot. A start is required to migrate the instance to healthy hardware.
The instance must be migrated to a new, healthy host by stopping and starting the instance. You can wait for Amazon EC2 to perform the stop and start of your instance. Or, you can manually stop and start the instance to migrate it to a new, healthy host.
When the instance is in the running state, choose Actions, Monitor and troubleshoot, Get system log.
https://forums.aws.amazon.com/thread.jspa?threadID=84461
By default EC2 instances are not redundant, Redundancy is achieved through using our other services such as Auto-scaling and Elastic lad balancing where you can mitigate around instance failure and make your application resilient of the components that it's running on.
You have to construct your server stack in a way that you don't care of the entire server goes away. In other words, keep your data in non-instance storage (EBS volumes, RDS, SimpleDB, S3). You have to have the mentality that your "instance" can go away at any time and it won't bother you, because you can click once and start up another one.
https://aws.amazon.com/ec2/instance-types/
https://towardsdatascience.com/choosing-the-right-gpu-for-deep-learning-on-aws-d69c157d8c86
Copy Top 'Latest Modified 10 Files" from Linux Server and Download it to Local Laptop
https://stackoverflow.com/questions/16886179/scp-or-sftp-copy-multiple-files-with-single-command
https://stackoverflow.com/questions/1706882/get-the-date-a-day-before-current-time-in-bash
---------------------------------------------------------------------------------------
ssh kk@10.102.20.43
---------------------------------------------------------------------------------------
cd /efs/datadrive/app-data/vea/output/reports
---------------------------------------------------------------------------------------
ls -1t | head -8 # Its One 1 , not "L"
---------------------------------------------------------------------------------------
ls -1t | head -8 | tail -7 # date +%d-%m-%Y
---------------------------------------------------------------------------------------
date +%d-%m-%Y -d "2 day ago" # date --date='-2 day'
---------------------------------------------------------------------------------------
mkdir -p # xargs date +%d-%m-%Y -d "2 day"
---------------------------------------------------------------------------------------
zip
---------------------------------------------------------------------------------------
scp 703250313@10.102.20.43:/efs/datadrive/app-data/vea/output/reports/21-6-2021.csv .
---------------------------------------------------------------------------------------
Sunday, June 20, 2021
Unix Libraries - for windows (Alternatives) : GitBash and GOW(GNU on Windows)
I have used - Git Bash, and GOW (GNU on Windows)
BEST COLLECTIONS OF UNIX LIBRARIES FOR WINDOWS |
---|
Azure - Pipeline - Add Approver for Stage
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass
-
https://www.baeldung.com/spring-properties-file-outside-jar https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-featu...
-
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass
-
The decision was made to block such external HTTP repositories by default https://stackoverflow.com/questions/66980047/maven-build-failure-d...