你的位置:首页 > Java教程

[Java教程]在MacOs上配置Hadoop和Spark环境

在MacOs上配置hadoop和spark环境

Setting up Hadoop with Spark on MacOs

Instructions

  1. 准备环境
    如果没有brew,先google怎样安装brew
    先uninstall老版本的Hadoop

    brew cleanup hadoop

    然后更新homebrew formulae

    brew update
    brew upgrade
    brew cleanup

    检查版本信息

    brew info hadoop
    brew info apache-spark
    brew info sbt
    brew info scala

    如果以上程序没有安装,需要使用brew install app 进行安装。

  2. 安装环境 安装hadoop

    brew install hadoop

    安装spark

    brew install apache-spark scala sbt

  3. 设置环境变量
    使用vim编辑~/.bash_profile,将以下内容贴到最后

     # set environment variables  export JAVA_HOME=$(/usr/libexec/java_home)   export HADOOP_HOME=/usr/local/Cellar/hadoop/2.5.1   export HADOOP_CONF_DIR=$HADOOP_HOME/libexec/etc/hadoop  export SCALA_HOME=/usr/local/Cellar/apache-spark/1.1.0  # set path variables  export PATH=$PATH:$HADOOP_HOME/bin:$SCALA_HOME/bin   # set alias start & stop scripts  alias hstart=$HADOOP_HOME/sbin/start-dfs.sh;$HADOOP_HOME/sbin/start-yarn.sh  alias hstop=$HADOOP_HOME/sbin/stop-dfs.sh;$HADOOP_HOME/sbin/stop-yarn.sh
  4. Hadoop必须要使ssh生效,设置ssh

    • 配置文件路径:

      /etc/sshd_config

    • 生成秘钥:

      sh-3.2# sudo ssh-keygen -t rsa

       Generating public/private rsa key pair. Enter file in which to save the key (/var/root/.ssh/id_rsa): 输入/var/root/.ssh/id_rsa Enter passphrase (empty for no passphrase): [直接回车] Enter same passphrase again: [直接回车] Your identification has been saved in /var/root/.ssh/id_rsa. Your public key has been saved in /var/root/.ssh/id_rsa.pub. key fingerprint is: 97:e9:5a:5e:91:52:30:63:9e:34:1a:6f:24:64:75:af [email protected] The key's randomart image is: +--[ RSA 2048]----+ |    .=.X .  | |    . X B .  | |    . = . . | |     . + o  | |    S = E  | |     o . .  | |     o .  | |     + .   | |    . .   | +-----------------+
    • 修改配置文

      sudo vim /etc/ssh/sshd_config

       Port 22 #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress :: # The default requires explicit activation of protocol 1 Protocol 2 # HostKey for protocol version 1 #HostKey /etc/ssh/ssh_host_key # HostKeys for protocol version 2 #HostKey /etc/ssh/ssh_host_rsa_key #HostKey /etc/ssh/ssh_host_dsa_key #HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /var/root/.ssh/id_rsa # Lifetime and size of ephemeral version 1 server key KeyRegenerationInterval 1h ServerKeyBits 1024 # Logging # obsoletes QuietMode and FascistLogging SyslogFacility AUTHPRIV #LogLevel INFO # Authentication: LoginGraceTime 2m PermitRootLogin yes StrictModes yes #MaxAuthTries 6 #MaxSessions 10 RSAAuthentication yes PubkeyAuthentication yes
    • 启动ssh服务

      which sshd //查找sshd的位置。

      Mac 上sshd的位置在 /usr/sbin/sshd

      在终端输入sudo /usr/sbin/sshd即可启动sshd服务。

      ssh-keygen -t rsa
      cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

  5. 配置Hadoop
    到hadoop的安装路径

    cd usr/local/Cellar/hadoop/2.5.1/libexec/

    编辑etc/hadoop/hadoop-env.sh

     # this fixes the "scdynamicstore" warning  export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" 

    编辑etc/hadoop/core-site.

     <configuration>   <property>     <name>fs.defaultFS</name>     <value>hdfs://localhost:9000</value>   </property> </configuration>

    编辑etc/hadoop/hdfs-site.

     <configuration>   <property>     <name>dfs.replication</name>     <value>1</value>   </property> </configuration>

    编辑etc/hadoop/mapred-site.

     <configuration>   <property>     <name>mapreduce.framework.name</name>     <value>yarn</value>   </property> </configuration>

    编辑etc/hadoop/yarn-site.

     <configuration>   <property>     <name>yarn.nodemanager.aux-services</name>     <value>mapreduce_shuffle</value>   </property> </configuration>
  6. 开始启用Hadoop
    移动到Hadoop的root directory

    cd /usr/local/Cellar/hadoop/2.5.1

    格式化Hadoop HDFS

    ./bin/hdfs namenode -format

    启动NameNode和DataNode daemon

    ./sbin/start-dfs.sh

    从网页中查看

    http://localhost:50070/

    启动ResourceManager和NodeManager daemon

    ./sbin/start-yarn.sh

    检查所有的守护线程是不是已经在运行

    jps

    从网页中查看ResourceManager

    http://localhost:8088/

    创建HDFS目录

    ./bin/hdfs dfs -mkdir -p /user/{username}

    启动一个MapReduce的例子

     \#calculate pi  ./bin/hadoop jar libexec/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.1.jar pi 10 100
  7. 启动spark

    到Spark的安装目录

    cd /usr/local/Cellar/apache-spark/1.1.0

    启动Spark的例子

    ./bin/run-example SparkPi

    在网页中查看Spark任务

    http://localhost:4040/

    也可以使用Spark-submit来提交任务

     # pattern to launch an application in yarn-cluster mode ./bin/spark-submit --class <path.to.class> --master yarn-cluster [options] <app.jar> [options] # run example application (calculate pi) ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster libexec/lib/spark-examples-*.jar
  8. 结束