hadoop standalone docker + localhost spark client 설정

설치 순서는 이렇다.

hadoop + hive + spark + elasticsearch-hadoop 을 설치하고 설정

1. hadoop standlone 으로 설치 https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
2. hive 설치 http://mirror.navercorp.com/apache/hive/
3. spark 설치 https://spark.apache.org/downloads.html
4. spark 설정
   • $SPARK_HOME/conf/spark-env.sh : HADOOP_CONF_DIR=(하둡 설정파일 위치)
   • $SPARK_HOME/conf/spark-defaults.sh : spark.master yarn
   • conf/hive-site.xml 설정:

     <configuration>
           <property>
           <name>hive.metastore.warehouse.dir</name>
           <value>/usr/local/share/spark-2.4.3-bin-hadoop2.7</value>
           </property>
     </configuration>

5. elasticsearch-hadoop 다운로드 https://www.elastic.co/downloads/hadoop
6. elasticsearch-spark-*.jar 파일을 $SPARK_HOME/jars 로 복사
7. findspark 설치 : pip install findspark

주피터 노트북에서 이렇게 호출해서 쓴다.

import findspark
findspark.init()

from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession
esconf = [
   ( "es.nodes","10.10.78.195"),
   ( "es.port","9200")
]

conf = SparkConf().setAppName("PySpark App").setMaster("yarn")
for i in esconf:
    conf = conf.set(i[0],i[1])
spark = SparkSession.builder.config(conf=conf).getOrCreate()
df = spark.read.format("org.elasticsearch.spark.sql").load('es_index')
df.printSchema()