Thursday, October 12, 2023

Read CSV from S3

 import csv

def count_records(csv_file):
    record_count = 0
    first_line = None
    last_line = None

    # Open the CSV file and read it line by line
    with open(csv_file, 'r', newline='') as file:
        reader = csv.reader(file)
        try:
            first_line = next(reader)
            for row in reader:
                last_line = row
                record_count += 1
        except StopIteration:
            pass

    return record_count, first_line, last_line

# Specify the path to your CSV file
csv_file_path = 'path/to/your/file.csv'

# Count the records, get the first and last lines
num_records, first_row, last_row = count_records(csv_file_path)

# Display the results
print(f"Number of records in the CSV file: {num_records}")
print("First line:", first_row)
print("Last line:", last_row)

Tuesday, September 26, 2023

Kubernetes k8s

Day 3

1. INSTALLATION

kubeadm init  @@will be run on master, below step will be done as a part of this command.

1. pulling images of API server, etcd, scheduler, replication controller, network proxy

2. setting ca certificate for all the above components

3. writes controller 

4. start the pod of above components

5. gives user access token

6. give instruction to run kubectl cli to run.

7. gives join command to connect minion to master.

MASTER is the brain of our cluster

Following are key components in master:

  • API Server
  • Scheduler
  • Replication Controller
  • Etcd

 

Minion (Node)

  • POD

ReplicaSets

Deployments - declaritive for replicasets and pod. Eg rollout

SERVICES: Abstract access of container and pod using kubeproxy

DAEMONSETS

JOBS

CRONJOBS

CONFIGMAP

SECRETS

NETWORK POLICY

INGRESS

 

Monday, September 4, 2023

Docker Commands

Day - 1

yum install docker @@ install docker

systemctl restart docker @@ restart docker

systemctl enable docker @@ autostart docker if machine restart

docker version

docker images @@ List all the images in local

docker ps @@ list the running container 

docker ps -a @@ list all the container

docker run <image> @@ Starting a new container of image

docker run -i -t ubuntu /bin/bash @@ Give me interactive terminal of ubuntu image container

docker run -d smehta26/clock @@container run without log or detached

docker logs <containerId or containerName> @@print the logs

docker logs --tail 3 <containerId or containerName> @@print only 3 lines of logs

docker logs --tail 1 --follow <containerId or containerName> @@follow the logs

Starting container with user specific command

docker run -d ubuntu:14:04 sleep 1000000

Get inside the container

docker exec -it <containerId> bash

ps -eaf @@check the process once inside the container

docker exec -it <containerId> sh

Name the container

docker run -d --name c1 ubuntu:14.04 sleep 1000

docker run -d --name c2 ubuntu:14.04 sleep 1000

docker stop c1 @@graceful stop takes times

docker kill c2 @@forceful stop immediate

docker start c1

docker start c2

docker stop C1  @@Stop the container

docker start C1 - @@Start the container

docker kill C1 @@kill the container

docker rm C1 @@Remove the container

Inspect running container

docker inspect c1

Kill and remove all the running container

docker ps -q  @@give only running containerId

docker ps -aq @@gives all the containerId

docker kill $(docker ps -q); docker rm $(docker ps -aq)  @@kill all the running container & remove all container


CREATE YOUR OWN IMAGE:

open vi editor DockerFile and type below command

#FROM instruction is used ot define Base OS Image.

FROM ubuntu:14.04

#RUN instruction to execute command over previous step

RUN apt-get update

RUN apt-get install figlet

#CMD instruction to define default command which will run when container starts

CMD figlet zpaya

Save the editor

[root@master]# docker build -t figlet:1.0 .

[root@master]# docker images      

[root@master]# docker history figlet:1.0      

[root@master]# docker run figlet:1.0   

[root@master]# docker ps -a --no-trunc      

[root@master]# docker run figlet:1.0 figlet docker     

[root@master]# docker ps -a --no-trunc

Create a image with custome file:

open vi editor Customfile and type below command

FROM ubuntu:14.04

RUN apt-get update

RUN apt-get install figlet

#ENTRYPOINT instruction is there to define default command which will run when container starts

ENTRYPOINT ["figlet"]

Save the file and build the figlet:2.0 image and start the container with the image.


COPY:

open vi editor, copy from docker host to the container

COPY Customfile /etc/

COPY Dockerfile /tmp/

Save the file and build the figlet:3.0 image and start the container with the image.

[root@master-]# docker build -t figlet:3.0 -f abc1 .   

[root@master-]# docker run -it figlet:3.0 bash

root@81c9:/# cat /etc/abc    

root@81c9:/# cat /tmp/Dockerfile    

ENVIRONMENT VARIABLES:

vi Env

FROM ubuntu:14.04

ENV blog Blogger

ENV maven /usr/bin/mvn

ENV JAVA_HOME /usr/bin/java

CMD echo "$blog $maven $JAVA_HOME"

[root@master-]# docker build -t figlet:4.0 -f abc1 .   

[root@master-]# docker run -it figlet:4.0 bash


USER INSTRUCTION:

vi UserInstruction

FROM ubuntu:14.04

RUN useradd -d "/home/zpaya" -m -s "/bin/bash" zpaya

USER zpaya

CMD id

Save the file

[root@master-]# docker build -t figlet:5.0 -f abc1 .   

[root@master-]# docker run -it figlet:5.0 bash

EXPOSE PORT to INTERNET:

docker run -d -P nginx @@Expose the random port of nginx to Master node

docker run -d -p 1000:80 nginx @@Expose specified port of nginx(80) to masternode

ifconfig eth0 @@will give you the public.

http://139.xx.73.xx:1000/


Day - 2

1. Exercise:
Create Image of CentOS7 
Install Apache Webserver
Access webpage from Internet
Steps:
Create a folder myapache
vi Dockerfile  >> Paste below commands
FROM centos:7
RUN  yum install httpd -y
EXPOSE 80/tcp
CMD /usr/sbin/httpd -D FOREGROUND
SAVE the file
docker build -t myapache .
docker run -d -p 2100:80 myapache
NOW access the apache page from ip:port from internet



2. NETWORKING
docker network ls @@list the docker network
docker network create dev @@create a dev network with bridge driver
ifconfig br-NETWORKID @@get ip of created network
ifconfig docker0 @@for default bridge network
docker run -d --net dev --name webserver nginx  @
docker ps @@list the running container
docker run -ti --net dev alpine sh @@start container and start bone shell

docker network inspect dev

docker kill $(docker ps -aq)

3. Connect Frontend to Database
docker run -d --net dev --name gunicorn -P jpetazzo/trainingwheels
docker logs gunicorn @@error connecting db redis
docker run -d --name redis --net dev redis:alpine
docker ps

now access the page on internet

4. Upload image to docker hub
docker tag jpetazzo/trainingwheels zpaya/doublegunicorn
docker login
docker push zpaya/doublegunicorn
docker logout

docker save -o mywebimage.tar mywebimage
docker load -i mywebimage.tar

docker network ls
docker network rm dev

5. Accessing the Data Store or files from host to different containers:
Create a file in DockerHost

mkdir folder1; cd folder1

echo "This is file1" > file1

docker run -it -v /root/folder1:/tmp/sharedfolder ubuntu:14.04 bash @@This will create ubuntu container and open terminal in it.

open /tmp/sharedfolder  @you will be able to see all files and able to edit

docker run -it -v /root/folder1:/tmp/sharedfolder:ro centos:7 bash @@This will create centos container and open terminal in it. shared folder will be read only.

6. Connect and disconnect network to the container i.e. having multiple ips to container
docker network create dev
docker network connect dev container1
docker network disconnect dev ccontainer2
docker network rm dev







docker pull Y - download the container

docker push Z - upload the container

Code(Dockerfile) —> 

compile(docker build) —> 

DockerImage

Read CSV from S3

 import csv def count_records(csv_file):     record_count = 0     first_line = None     last_line = None     # Open the CSV file and read it...