成人免费xxxxx在线视频软件_久久精品久久久_亚洲国产精品久久久_天天色天天色_亚洲人成一区_欧美一级欧美三级在线观看

七張圖入門 Hadoop 在 K8S 環境中部署

開源
Hadoop 是 Apache 軟件基金會下一個開源分布式計算平臺，以 HDFS（Hadoop Distributed File System）、MapReduce（Hadoop2.0 加入了 YARN，Yarn 是資源調度框架，能夠細粒度的管理和調度任務，還能夠支持其他的計算框架，比如 spark）。

一、概述

為核心的 Hadoop 為用戶提供了系統底層細節透明的分布式基礎架構。hdfs 的高容錯性、高伸縮性、高效性等優點讓用戶可以將 Hadoop 部署在低廉的硬件上，形成分布式系統。目前最新版本已經是 3.x 了，官方文檔：https://hadoop.apache.org/docs/current/index.html

HDFS

YARN

二、開始部署

1）添加源

地址：??https://artifacthub.io/packages/helm/apache-hadoop-helm/hadoop??

helm repo add apache-hadoop-helm https://pfisterer.github.io/apache-hadoop-helm/
helm pull apache-hadoop-helm/hadoop --version 1.2.0
tar -xf hadoop-1.2.0.tgz

2）構建鏡像 Dockerfile 

FROM myharbor.com/bigdata/centos:7.9.2009
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
RUN export LANG=zh_CN.UTF-8
# 創建用戶和用戶組，跟yaml編排里的spec.template.spec.containers. securityContext.runAsUser: 9999
RUN groupadd --system --gid=9999 admin && useradd --system --home-dir /home/admin --uid=9999 --gid=admin admin
# 安裝sudo
RUN yum -y install sudo ; chmod 640 /etc/sudoers
# 給admin添加sudo權限
RUN echo "admin ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN yum -y install install net-tools telnet wget
RUN mkdir /opt/apache/
ADD jdk-8u212-linux-x64.tar.gz /opt/apache/
ENV JAVA_HOME=/opt/apache/jdk1.8.0_212
ENV PATH=$JAVA_HOME/bin:$PATH
ENV HADOOP_VERSION 3.3.2
ENV HADOOP_HOME=/opt/apache/hadoop
ENV HADOOP_COMMON_HOME=${HADOOP_HOME} \
    HADOOP_HDFS_HOME=${HADOOP_HOME} \
    HADOOP_MAPRED_HOME=${HADOOP_HOME} \
    HADOOP_YARN_HOME=${HADOOP_HOME} \
    HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop \
    PATH=${PATH}:${HADOOP_HOME}/bin
#RUN curl --silent --output /tmp/hadoop.tgz https://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz && tar --directory /opt/apache -xzf /tmp/hadoop.tgz && rm /tmp/hadoop.tgz
ADD hadoop-${HADOOP_VERSION}.tar.gz /opt/apache
RUN ln -s /opt/apache/hadoop-${HADOOP_VERSION} ${HADOOP_HOME}
RUN chown -R admin:admin /opt/apache
WORKDIR $HADOOP_HOME
# Hdfs ports
EXPOSE 50010 50020 50070 50075 50090 8020 9000
# Mapred ports
EXPOSE 19888
#Yarn ports
EXPOSE 8030 8031 8032 8033 8040 8042 8088
#Other ports
EXPOSE 49707 2122

開始構建鏡像

docker build -t myharbor.com/bigdata/hadoop:3.3.2 . --no-cache
### 參數解釋
# -t：指定鏡像名稱
# . ：當前目錄Dockerfile
# -f：指定Dockerfile路徑
#  --no-cache：不緩存

推送到鏡像倉庫 

docker push myharbor.com/bigdata/hadoop:3.3.2

調整目錄結構 

mkdir hadoop/templates/hdfs hadoop/templates/yarn
mv hadoop/templates/hdfs-* hadoop/templates/hdfs/
mv hadoop/templates/yarn-* hadoop/templates/yarn/

3）修改配置

  • hadoop/values.yaml

image:
  repository: myharbor.com/bigdata/hadoop
  tag: 3.3.2
  pullPolicy: IfNotPresent
...
persistence:
  nameNode:
    enabled: true
    storageClass: "hadoop-nn-local-storage"
    accessMode: ReadWriteOnce
    size: 10Gi
    local:
    - name: hadoop-nn-0
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/hadoop/nn/data/data1"
  dataNode:
    enabled: true
    storageClass: "hadoop-dn-local-storage"
    accessMode: ReadWriteOnce
    size: 20Gi
    local:
    - name: hadoop-dn-0
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/hadoop/dn/data/data1"
    - name: hadoop-dn-1
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/hadoop/dn/data/data2"
    - name: hadoop-dn-2
      host: "local-168-182-110"
      path: "/opt/bigdata/servers/hadoop/dn/data/data3"
    - name: hadoop-dn-3
      host: "local-168-182-111"
      path: "/opt/bigdata/servers/hadoop/dn/data/data1"
    - name: hadoop-dn-4
      host: "local-168-182-111"
      path: "/opt/bigdata/servers/hadoop/dn/data/data2"
    - name: hadoop-dn-5
      host: "local-168-182-111"
      path: "/opt/bigdata/servers/hadoop/dn/data/data3"
    - name: hadoop-dn-6
      host: "local-168-182-112"
      path: "/opt/bigdata/servers/hadoop/dn/data/data1"
    - name: hadoop-dn-7
      host: "local-168-182-112"
      path: "/opt/bigdata/servers/hadoop/dn/data/data2"
   - name: hadoop-dn-8
      host: "local-168-182-112"
      path: "/opt/bigdata/servers/hadoop/dn/data/data3"
...
service:
  nameNode:
    type: NodePort
    ports:
      dfs: 9000
      webhdfs: 9870
    nodePorts:
      dfs: 30900
      webhdfs: 30870
  dataNode:
    type: NodePort
    ports:
      dfs: 9000
      webhdfs: 9864
    nodePorts:
      dfs: 30901
      webhdfs: 30864
  resourceManager:
    type: NodePort
    ports:
      web: 8088
    nodePorts:
      web: 30088
...
securityContext:
  runAsUser: 9999
  privileged: true

  • hadoop/templates/hdfs/hdfs-nn-pv.yaml

{{- range .Values.persistence.nameNode.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.persistence.nameNode.storageClass }}
  capacity:
    storage: {{ $.Values.persistence.nameNode.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

  • hadoop/templates/hdfs/hdfs-dn-pv.yaml

{{- range .Values.persistence.dataNode.local }}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .name }}
  labels:
    name: {{ .name }}
spec:
  storageClassName: {{ $.Values.persistence.dataNode.storageClass }}
  capacity:
    storage: {{ $.Values.persistence.dataNode.size }}
  accessModes:
    - ReadWriteOnce
  local:
    path: {{ .path }}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - {{ .host }}
---
{{- end }}

  • 修改 hdfs service

mv hadoop/templates/hdfs/hdfs-nn-svc.yaml hadoop/templates/hdfs/hdfs-nn-svc-headless.yaml
mv hadoop/templates/hdfs/hdfs-dn-svc.yaml hadoop/templates/hdfs/hdfs-dn-svc-headless.yaml
# 注意修改名稱，不要重復

hadoop/templates/hdfs/hdfs-nn-svc.yaml 

# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: {{ include "hadoop.fullname" . }}-hdfs-nn
  labels:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    helm.sh/chart: {{ include "hadoop.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: dfs
    port: {{ .Values.service.nameNode.ports.dfs }}
    protocol: TCP
    nodePort: {{ .Values.service.nameNode.nodePorts.dfs }}
  - name: webhdfs
    port: {{ .Values.service.nameNode.ports.webhdfs }}
    nodePort: {{ .Values.service.nameNode.nodePorts.webhdfs }}
  type: {{ .Values.service.nameNode.type }}
  selector:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: hdfs-nn

hadoop/templates/hdfs/hdfs-dn-svc.yaml 

# A headless service to create DNS records
apiVersion: v1
kind: Service
metadata:
  name: {{ include "hadoop.fullname" . }}-hdfs-dn
  labels:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    helm.sh/chart: {{ include "hadoop.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: hdfs-nn
spec:
  ports:
  - name: dfs
    port: {{ .Values.service.dataNode.ports.dfs }}
    protocol: TCP
    nodePort: {{ .Values.service.dataNode.nodePorts.dfs }}
  - name: webhdfs
    port: {{ .Values.service.dataNode.ports.webhdfs }}
    nodePort: {{ .Values.service.dataNode.nodePorts.webhdfs }}
  type: {{ .Values.service.dataNode.type }}
  selector:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: hdfs-dn

  • 修改 yarn service

mv hadoop/templates/yarn/yarn-nm-svc.yaml hadoop/templates/yarn/yarn-nm-svc-headless.yaml
mv hadoop/templates/yarn/yarn-rm-svc.yaml hadoop/templates/yarn/yarn-rm-svc-headless.yaml
mv hadoop/templates/yarn/yarn-ui-svc.yaml hadoop/templates/yarn/yarn-rm-svc.yaml
# 注意修改名稱，不要重復

hadoop/templates/yarn/yarn-rm-svc.yaml 

# Service to access the yarn web ui
apiVersion: v1
kind: Service
metadata:
  name: {{ include "hadoop.fullname" . }}-yarn-rm
  labels:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    helm.sh/chart: {{ include "hadoop.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: yarn-rm
spec:
  ports:
  - port: {{ .Values.service.resourceManager.ports.web }}
    name: web
    nodePort: {{ .Values.service.resourceManager.nodePorts.web }}
  type: {{ .Values.service.resourceManager.type }}
  selector:
    app.kubernetes.io/name: {{ include "hadoop.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/component: yarn-rm

  • 修改控制器

在所有控制中新增如下內容： 

containers:
...
  securityContext:
    runAsUser: {{ .Values.securityContext.runAsUser }}
    privileged: {{ .Values.securityContext.privileged }}

  • hadoop/templates/hadoop-configmap.yaml

### 1、將/root換成/opt/apache
### 2、TMP_URL="http://{{ include "hadoop.fullname" . }}-yarn-rm-headless:8088/ws/v1/cluster/info"

4）開始安裝 

# 創建存儲目錄
mkdir -p /opt/bigdata/servers/hadoop/{nn,dn}/data/data{1..3}
helm install hadoop ./hadoop -n hadoop --create-namespace

NOTES 

NAME: hadoop
LAST DEPLOYED: Sat Sep 24 17:00:55 2022
NAMESPACE: hadoop
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. You can check the status of HDFS by running this command:
   kubectl exec -n hadoop -it hadoop-hadoop-hdfs-nn-0 -- /opt/hadoop/bin/hdfs dfsadmin -report
2. You can list the yarn nodes by running this command:
   kubectl exec -n hadoop -it hadoop-hadoop-yarn-rm-0 -- /opt/hadoop/bin/yarn node -list
3. Create a port-forward to the yarn resource manager UI:
   kubectl port-forward -n hadoop hadoop-hadoop-yarn-rm-0 8088:8088
   Then open the ui in your browser:
   open http://localhost:8088
4. You can run included hadoop tests like this:
   kubectl exec -n hadoop -it hadoop-hadoop-yarn-nm-0 -- /opt/hadoop/bin/hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.2-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt
5. You can list the mapreduce jobs like this:
   kubectl exec -n hadoop -it hadoop-hadoop-yarn-rm-0 -- /opt/hadoop/bin/mapred job -list
6. This chart can also be used with the zeppelin chart
    helm install --namespace hadoop --set hadoop.useConfigMap=true,hadoop.configMapName=hadoop-hadoop stable/zeppelin
7. You can scale the number of yarn nodes like this:
   helm upgrade hadoop --set yarn.nodeManager.replicas=4 stable/hadoop
   Make sure to update the values.yaml if you want to make this permanent.

查看 

kubectl get pods,svc -n hadoop -owide

hdfs web：??http://192.168.182.110:30870/??

yarn web：??http://192.168.182.110:30088/??

5）測試驗證

HDFS 測試驗證 

kubectl exec -it hadoop-hadoop-hdfs-nn-0 -n hadoop -- bash
[root@local-168-182-110 hadoop]# kubectl exec -it hadoop-hadoop-hdfs-nn-0 -n hadoop -- bash
bash-4.2$
bash-4.2$
bash-4.2$ hdfs dfs -mkdir /tmp
bash-4.2$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - admin supergroup          0 2022-09-24 17:56 /tmp
bash-4.2$ echo "test hadoop" >test.txt
bash-4.2$ hdfs dfs -put test.txt /tmp/
bash-4.2$ hdfs dfs -ls /tmp/
Found 1 items
-rw-r--r--   3 admin supergroup         12 2022-09-24 17:57 /tmp/test.txt
bash-4.2$ hdfs dfs -cat /tmp/
cat: `/tmp': Is a directory
bash-4.2$ hdfs dfs -cat /tmp/test.txt
test hadoop
bash-4.2$

Yarn 的測試驗證等后面講到 hive on k8s 再來測試驗證。

6）卸載 

helm uninstall hadoop -n hadoop
kubectl delete pod -n hadoop `kubectl get pod -n hadoop|awk 'NR>1{print $1}'` --force
kubectl patch ns hadoop -p '{"metadata":{"finalizers":null}}'
kubectl delete ns hadoop --force

這里也提供 git 下載地址，有需要的小伙伴可以下載部署玩玩：??https://gitee.com/hadoop-bigdata/hadoop-on-k8s??

????在 k8s 集群中 yarn 會慢慢被弱化，直接使用 k8s 資源調度，而不再使用 yarn 去調度資源了，這里只是部署了單點，僅限于測試環境使用，下一篇文章會講 Hadoop 高可用 on k8s 實現，請小伙伴耐心等待。

責任編輯：龐桂玉 來源： 馬哥Linux運維
HadoopK8S開源
相關推薦

2023-09-06 08:12:04

k8s云原生

2023-07-11 07:12:21

Hadoop部署mysql

2023-07-10 07:22:16

2023-12-25 08:05:35

2020-05-12 10:20:39

K8s kubernetes中間件

2024-01-29 13:03:02

2022-01-22 21:38:19

K8SRedisRedis clus

2021-11-29 07:47:56

RocketMQ分布式消息

2024-01-12 08:03:29

k8s配置持久化

2021-09-26 07:43:08

KongKongaK8s

2022-04-22 13:32:01

K8s容器引擎架構

2021-08-13 07:00:41

云原生k8sspringboot

2021-08-26 07:20:05

云原生K8sSpringboot

2023-09-08 08:09:12

k8sservice服務

2021-11-04 07:49:58

K8SStatefulSetMySQL

2022-01-02 08:42:50

架構部署容器

2023-02-27 07:40:00

2023-11-06 07:16:22

WasmK8s模塊

2024-05-20 15:39:00

Karmada混合云多云

2021-05-07 09:31:33

KindK8s Operator
點贊
收藏

51CTO技術棧公眾號

主站蜘蛛池模板： 国产黄视频在线播放 | 国产日韩久久 | 亚洲一区二区三区在线视频 | 免费网站国产 | 一本一道久久a久久精品蜜桃 | 伊人超碰在线 | 成年人的视频免费观看 | 亚洲欧美激情精品一区二区 | 国产精品久久在线 | 日韩一区二区在线播放 | 久久精品这里精品 | 日韩欧美专区 | 三级免费 | 青青草av | 成av人电影在线 | 精品视频在线播放 | 91精品国产91久久久久久吃药 | 久久久久久久久久爱 | 亚洲综合无码一区二区 | 久久乐国产精品 | 婷婷综合激情 | 久久精品国产免费看久久精品 | 免费激情 | 欧美电影免费观看 | 成人国产精品久久久 | 日韩在线一区二区三区 | 亚洲第一av | 久久久成人免费一区二区 | av看片网站 | 欧美一区二区三区视频 | 一本一道久久a久久精品综合蜜臀 | 亚洲日日操| 奇米四色在线观看 | 欧美激情久久久久久 | 久久久久久精 | 精品国产一区二区三区久久久蜜月 | 色先锋影音| 亚洲高清视频在线观看 | 91精品国产麻豆 | 91高清在线视频 | 久久亚洲一区二区 |