Read CSV from S3

 import csv

def count_records(csv_file):
    record_count = 0
    first_line = None
    last_line = None

    # Open the CSV file and read it line by line
    with open(csv_file, 'r', newline='') as file:
        reader = csv.reader(file)
        try:
            first_line = next(reader)
            for row in reader:
                last_line = row
                record_count += 1
        except StopIteration:
            pass

    return record_count, first_line, last_line

# Specify the path to your CSV file
csv_file_path = 'path/to/your/file.csv'

# Count the records, get the first and last lines
num_records, first_row, last_row = count_records(csv_file_path)

# Display the results
print(f"Number of records in the CSV file: {num_records}")
print("First line:", first_row)
print("Last line:", last_row)

Kubernetes k8s

Day 3

1. INSTALLATION

kubeadm init  @@will be run on master, below step will be done as a part of this command.

1. pulling images of API server, etcd, scheduler, replication controller, network proxy

2. setting ca certificate for all the above components

3. writes controller 

4. start the pod of above components

5. gives user access token

6. give instruction to run kubectl cli to run.

7. gives join command to connect minion to master.

MASTER is the brain of our cluster

Following are key components in master:

• API Server
• Scheduler
• Replication Controller
• Etcd

 

Minion (Node)

• POD

ReplicaSets

Deployments - declaritive for replicasets and pod. Eg rollout

SERVICES: Abstract access of container and pod using kubeproxy

DAEMONSETS

JOBS

CRONJOBS

CONFIGMAP

SECRETS

NETWORK POLICY

INGRESS

 

Docker Commands

Day - 1

yum install docker @@ install docker

systemctl restart docker @@ restart docker

systemctl enable docker @@ autostart docker if machine restart

docker version

docker images @@ List all the images in local

docker ps @@ list the running container 

docker ps -a @@ list all the container

docker run <image> @@ Starting a new container of image

docker run -i -t ubuntu /bin/bash @@ Give me interactive terminal of ubuntu image container

docker run -d smehta26/clock @@container run without log or detached

docker logs <containerId or containerName> @@print the logs

docker logs --tail 3 <containerId or containerName> @@print only 3 lines of logs

docker logs --tail 1 --follow <containerId or containerName> @@follow the logs

Starting container with user specific command

docker run -d ubuntu:14:04 sleep 1000000

Get inside the container

docker exec -it <containerId> bash

ps -eaf @@check the process once inside the container

docker exec -it <containerId> sh

Name the container

docker run -d --name c1 ubuntu:14.04 sleep 1000

docker run -d --name c2 ubuntu:14.04 sleep 1000

docker stop c1 @@graceful stop takes times

docker kill c2 @@forceful stop immediate

docker start c1

docker start c2

docker stop C1  @@Stop the container

docker start C1 - @@Start the container

docker kill C1 @@kill the container

docker rm C1 @@Remove the container

Inspect running container

docker inspect c1

Kill and remove all the running container

docker ps -q  @@give only running containerId

docker ps -aq @@gives all the containerId

docker kill $(docker ps -q); docker rm $(docker ps -aq)  @@kill all the running container & remove all container

CREATE YOUR OWN IMAGE:

open vi editor DockerFile and type below command

#FROM instruction is used ot define Base OS Image.

FROM ubuntu:14.04

#RUN instruction to execute command over previous step

RUN apt-get update

RUN apt-get install figlet

#CMD instruction to define default command which will run when container starts

CMD figlet zpaya

Save the editor

[root@master]# docker build -t figlet:1.0 .

[root@master]# docker images      

[root@master]# docker history figlet:1.0      

[root@master]# docker run figlet:1.0   

[root@master]# docker ps -a --no-trunc      

[root@master]# docker run figlet:1.0 figlet docker     

[root@master]# docker ps -a --no-trunc

Create a image with custome file:

open vi editor Customfile and type below command

FROM ubuntu:14.04

RUN apt-get update

RUN apt-get install figlet

#ENTRYPOINT instruction is there to define default command which will run when container starts

ENTRYPOINT ["figlet"]

Save the file and build the figlet:2.0 image and start the container with the image.

COPY:

open vi editor, copy from docker host to the container

COPY Customfile /etc/

COPY Dockerfile /tmp/

Save the file and build the figlet:3.0 image and start the container with the image.

[root@master-]# docker build -t figlet:3.0 -f abc1 .   

[root@master-]# docker run -it figlet:3.0 bash

root@81c9:/# cat /etc/abc    

root@81c9:/# cat /tmp/Dockerfile    

ENVIRONMENT VARIABLES:

vi Env

FROM ubuntu:14.04

ENV blog Blogger

ENV maven /usr/bin/mvn

ENV JAVA_HOME /usr/bin/java

CMD echo "$blog $maven $JAVA_HOME"

[root@master-]# docker build -t figlet:4.0 -f abc1 .   

[root@master-]# docker run -it figlet:4.0 bash

USER INSTRUCTION:

vi UserInstruction

FROM ubuntu:14.04

RUN useradd -d "/home/zpaya" -m -s "/bin/bash" zpaya

USER zpaya

CMD id

Save the file

[root@master-]# docker build -t figlet:5.0 -f abc1 .   

[root@master-]# docker run -it figlet:5.0 bash

EXPOSE PORT to INTERNET:

docker run -d -P nginx @@Expose the random port of nginx to Master node

docker run -d -p 1000:80 nginx @@Expose specified port of nginx(80) to masternode

ifconfig eth0 @@will give you the public.

http://139.xx.73.xx:1000/

Day - 2

1. Exercise:

Create Image of CentOS7 

Install Apache Webserver

Access webpage from Internet

Steps:

Create a folder myapache

vi Dockerfile  >> Paste below commands

FROM centos:7

RUN  yum install httpd -y

EXPOSE 80/tcp

CMD /usr/sbin/httpd -D FOREGROUND

SAVE the file

docker build -t myapache .

docker run -d -p 2100:80 myapache

NOW access the apache page from ip:port from internet

2. NETWORKING

docker network ls @@list the docker network

docker network create dev @@create a dev network with bridge driver

ifconfig br-NETWORKID @@get ip of created network

ifconfig docker0 @@for default bridge network

docker run -d --net dev --name webserver nginx  @

docker ps @@list the running container

docker run -ti --net dev alpine sh @@start container and start bone shell

docker network inspect dev

docker kill $(docker ps -aq)

3. Connect Frontend to Database

docker run -d --net dev --name gunicorn -P jpetazzo/trainingwheels

docker logs gunicorn @@error connecting db redis

docker run -d --name redis --net dev redis:alpine

docker ps

now access the page on internet

4. Upload image to docker hub

docker tag jpetazzo/trainingwheels zpaya/doublegunicorn

docker login

docker push zpaya/doublegunicorn

docker logout

docker save -o mywebimage.tar mywebimage

docker load -i mywebimage.tar

docker network ls

docker network rm dev

5. Accessing the Data Store or files from host to different containers:

Create a file in DockerHost

mkdir folder1; cd folder1

echo "This is file1" > file1

docker run -it -v /root/folder1:/tmp/sharedfolder ubuntu:14.04 bash @@This will create ubuntu container and open terminal in it.

open /tmp/sharedfolder  @you will be able to see all files and able to edit

docker run -it -v /root/folder1:/tmp/sharedfolder:ro centos:7 bash @@This will create centos container and open terminal in it. shared folder will be read only.

6. Connect and disconnect network to the container i.e. having multiple ips to container

docker network create dev

docker network connect dev container1

docker network disconnect dev ccontainer2

docker network rm dev

docker pull Y - download the container

docker push Z - upload the container

Code(Dockerfile) —> 

compile(docker build) —> 

DockerImage