[hjr@master bin]$ ./spark-shell Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true 16/03/31 13:56:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/03/31 13:56:13 INFO spark.SecurityManager: Changing view acls to: hjr 16/03/31 13:56:13 INFO spark.SecurityManager: Changing modify acls to: hjr 16/03/31 13:56:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr) 16/03/31 13:56:13 INFO spark.HttpServer: Starting HTTP Server 16/03/31 13:56:13 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/03/31 13:56:13 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:39368 16/03/31 13:56:13 INFO util.Utils: Successfully started service 'HTTP class server' on port 39368. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.1 /_/ Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80) Type in expressions to have them evaluated. Type :help for more information. 16/03/31 13:56:22 INFO spark.SparkContext: Running Spark version 1.4.1 16/03/31 13:56:22 WARN spark.SparkConf: SPARK_CLASSPATH was detected (set to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar'). This is deprecated in Spark 1.0+. Please instead use: - ./spark-submit with --driver-class-path to augment the driver classpath - spark.executor.extraClassPath to augment the executor classpath 16/03/31 13:56:22 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around. 16/03/31 13:56:22 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around. 16/03/31 13:56:22 INFO spark.SecurityManager: Changing view acls to: hjr 16/03/31 13:56:22 INFO spark.SecurityManager: Changing modify acls to: hjr 16/03/31 13:56:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr) 16/03/31 13:56:23 INFO slf4j.Slf4jLogger: Slf4jLogger started 16/03/31 13:56:23 INFO Remoting: Starting remoting 16/03/31 13:56:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.13.43:40169]
16/03/31 13:56:23 INFO util.Utils: Successfully started service 'sparkDriver' on port 40169.
16/03/31 13:56:23 INFO spark.SparkEnv: Registering MapOutputTracker
16/03/31 13:56:23 INFO spark.SparkEnv: Registering BlockManagerMaster
16/03/31 13:56:23 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-9e5228e8-d1d9-49ac-b187-0da05e0359e4/blockmgr-a77c56e4-b68d-42fa-95f9-c0e783daf7d0
16/03/31 13:56:23 INFO storage.MemoryStore: MemoryStore started with capacity 1069.1 MB
16/03/31 13:56:24 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-9e5228e8-d1d9-49ac-b187-0da05e0359e4/httpd-f9ee9698-2a94-4b33-b660-64cfa3e2e4f2
16/03/31 13:56:24 INFO spark.HttpServer: Starting HTTP Server
16/03/31 13:56:24 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/03/31 13:56:24 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:52432
16/03/31 13:56:24 INFO util.Utils: Successfully started service 'HTTP file server' on port 52432.
16/03/31 13:56:24 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/03/31 13:56:24 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/03/31 13:56:24 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/03/31 13:56:24 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/03/31 13:56:24 INFO ui.SparkUI: Started SparkUI at http://192.168.13.43:4040
16/03/31 13:56:25 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@master.casking.com:7077/user/Master...
16/03/31 13:56:25 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160331135625-0015
16/03/31 13:56:25 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 54303.
16/03/31 13:56:25 INFO netty.NettyBlockTransferService: Server created on 54303
16/03/31 13:56:25 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/03/31 13:56:25 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.43:54303 with 1069.1 MB RAM, BlockManagerId(driver, 192.168.13.43, 54303)
16/03/31 13:56:25 INFO storage.BlockManagerMaster: Registered BlockManager
16/03/31 13:56:26 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/03/31 13:56:26 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/03/31 13:56:28 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
java.lang.RuntimeException: java.io.IOException: Permission denied
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:330)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:116)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:163)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:161)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:168)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
at $iwC$$iwC.<init>(<console>:9)
at $iwC.<init>(<console>:18)
at <init>(<console>:20)
at .<init>(<console>:24)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:130)
at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2001)
at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:432)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:328)
... 56 more
<console>:10: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:10: error: not found: value sqlContext
import sqlContext.sql
^
scala> 查看Hadoop的core-site.xml文件的hadoop.tmp.dir的配置路径，并
修改其访问权限：
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/spark/hadoop-2.4.0/temp</value>
<description>Abase for other temporary directories.
</description>
</property>我的spark和hadoop安装配置都是使用root用户，所以，将新建的用户hjr新加到root组里面去，然后再修改hadoop.tmp.dir路径的权限：
hadoop/spark集群安装配置在root组：
[root@master spark-1.4.1-bin-hadoop2.4]# id
uid=0(root) gid=0(root) groups=0(root)
[root@master spark-1.4.1-bin-hadoop2.4]# 查看hjr用户所属组：
[root@master spark-1.4.1-bin-hadoop2.4]# su hjr
[hjr@master spark-1.4.1-bin-hadoop2.4]$ id
uid=1012(hjr) gid=1012(hjr) groups=1012(hjr)
[hjr@master spark-1.4.1-bin-hadoop2.4]$ 将用户hjr添加到root组里面去：
[hjr@master spark-1.4.1-bin-hadoop2.4]$ su root
Password:
[root@master spark-1.4.1-bin-hadoop2.4]# usermod -g root hjr
[root@master spark-1.4.1-bin-hadoop2.4]# su hjr
[hjr@master spark-1.4.1-bin-hadoop2.4]$ id
uid=1012(hjr) gid=0(root) groups=0(root)
[hjr@master spark-1.4.1-bin-hadoop2.4]$ 退出spark-shell来,发现问题了：
根目录下的/tmp文件夹的访问权限没有提上来：
scala> exit
warning: there were 1 deprecation warning(s); re-run with -deprecation for details
16/04/06 14:12:40 INFO spark.SparkContext: Invoking stop() from shutdown hook
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/04/06 14:12:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/04/06 14:12:40 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.13.43:4040
16/04/06 14:12:40 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/04/06 14:12:40 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors
16/04/06 14:12:40 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down
16/04/06 14:12:40 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/06 14:12:40 INFO util.Utils: path = /tmp/spark-4321496c-dcdc-4d6f-abe2-576d81b4a2fb/blockmgr-44be0eb3-a333-4f63-827d-05c926a3d3d1, already present as root for deletion.
16/04/06 14:12:40 INFO storage.MemoryStore: MemoryStore cleared
16/04/06 14:12:40 INFO storage.BlockManager: BlockManager stopped
16/04/06 14:12:40 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/04/06 14:12:40 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/06 14:12:40 INFO spark.SparkContext: Successfully stopped SparkContext
16/04/06 14:12:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/04/06 14:12:40 INFO util.Utils: Shutdown hook called
16/04/06 14:12:40 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-4321496c-dcdc-4d6f-abe2-576d81b4a2fb
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-b177a1a8-0c25-4475-8f06-dddb3a49fe08
16/04/06 14:12:40 INFO util.Utils: Deleting directory /tmp/spark-f731a0a4-c5cf-4db5-baab-ec3a4d28cf26切换到root用户下，将/tmp文件夹权限提上来就行：
chmod -R 777 /tmp
再次执行，妥妥的。。。。。。
[hjr@master bin]$ ./spark-shell
16/04/06 15:07:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/06 15:07:39 INFO spark.SecurityManager: Changing view acls to: hjr
16/04/06 15:07:39 INFO spark.SecurityManager: Changing modify acls to: hjr
16/04/06 15:07:39 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/04/06 15:07:40 INFO spark.HttpServer: Starting HTTP Server
16/04/06 15:07:40 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:40 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:51949
16/04/06 15:07:40 INFO util.Utils: Successfully started service 'HTTP class server' on port 51949.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.1
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
16/04/06 15:07:49 INFO spark.SparkContext: Running Spark version 1.4.1
16/04/06 15:07:49 WARN spark.SparkConf:
SPARK_CLASSPATH was detected (set to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
16/04/06 15:07:49 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/04/06 15:07:49 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to ':/home/spark/spark-1.4.1-bin-hadoop2.4/lib/mysql-connector-java-5.1.26-bin.jar' as a work-around.
16/04/06 15:07:49 INFO spark.SecurityManager: Changing view acls to: hjr
16/04/06 15:07:49 INFO spark.SecurityManager: Changing modify acls to: hjr
16/04/06 15:07:49 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hjr); users with modify permissions: Set(hjr)
16/04/06 15:07:50 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/04/06 15:07:50 INFO Remoting: Starting remoting
16/04/06 15:07:50 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.13.43:59810]
16/04/06 15:07:50 INFO util.Utils: Successfully started service 'sparkDriver' on port 59810.
16/04/06 15:07:50 INFO spark.SparkEnv: Registering MapOutputTracker
16/04/06 15:07:50 INFO spark.SparkEnv: Registering BlockManagerMaster
16/04/06 15:07:50 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-eb699480-b70b-4273-b8e9-433764a62c0e/blockmgr-22f931fa-c02a-409e-8acc-af6b4ce3f309
16/04/06 15:07:50 INFO storage.MemoryStore: MemoryStore started with capacity 267.3 MB
16/04/06 15:07:51 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-eb699480-b70b-4273-b8e9-433764a62c0e/httpd-d5ad4baa-989e-4731-9944-652645c86fef
16/04/06 15:07:51 INFO spark.HttpServer: Starting HTTP Server
16/04/06 15:07:51 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:51 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:47726
16/04/06 15:07:51 INFO util.Utils: Successfully started service 'HTTP file server' on port 47726.
16/04/06 15:07:51 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/04/06 15:07:51 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/04/06 15:07:51 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/04/06 15:07:51 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/04/06 15:07:51 INFO ui.SparkUI: Started SparkUI at http://192.168.13.43:4040
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@master.casking.com:7077/user/Master...
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160406150752-0004
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor added: app-20160406150752-0004/0 on worker-20160406135801-192.168.13.45-47341 (192.168.13.45:47341) with 1 cores
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160406150752-0004/0 on hostPort 192.168.13.45:47341 with 1 cores, 512.0 MB RAM
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor added: app-20160406150752-0004/1 on worker-20160406135801-192.168.13.44-60023 (192.168.13.44:60023) with 1 cores
16/04/06 15:07:52 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160406150752-0004/1 on hostPort 192.168.13.44:60023 with 1 cores, 512.0 MB RAM
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/1 is now LOADING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/0 is now LOADING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/0 is now RUNNING
16/04/06 15:07:52 INFO client.AppClient$ClientActor: Executor updated: app-20160406150752-0004/1 is now RUNNING
16/04/06 15:07:53 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51708.
16/04/06 15:07:53 INFO netty.NettyBlockTransferService: Server created on 51708
16/04/06 15:07:53 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/04/06 15:07:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.43:51708 with 267.3 MB RAM, BlockManagerId(driver, 192.168.13.43, 51708)
16/04/06 15:07:53 INFO storage.BlockManagerMaster: Registered BlockManager
16/04/06 15:07:53 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
16/04/06 15:07:53 INFO repl.SparkILoop: Created spark context..
Spark context available as sc.
16/04/06 15:07:55 INFO hive.HiveContext: Initializing execution hive, version 0.13.1
16/04/06 15:07:56 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.13.43:9083
16/04/06 15:07:56 INFO hive.metastore: Connected to metastore.
16/04/06 15:07:57 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.13.45:51876/user/Executor#820989273]) with ID 0
16/04/06 15:07:57 INFO cluster.SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@192.168.13.44:38556/user/Executor#-638549016]) with ID 1
16/04/06 15:07:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.44:35787 with 267.3 MB RAM, BlockManagerId(1, 192.168.13.44, 35787)
16/04/06 15:07:57 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.13.45:33428 with 267.3 MB RAM, BlockManagerId(0, 192.168.13.45, 33428)
16/04/06 15:07:58 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
16/04/06 15:07:58 INFO repl.SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala>