Wednesday, 15 April 2015

How to run Apache Spark on Windows7 in standalone mode

So far, we might have done setup of Spark with Hadoop, EC2 or mesos on Linux machine.  But what if we don’t want with Hadoop/EC2, we just want to run it in standalone mode on windows.
Here we’ll see how we can run Spark on Windows machine.

Prerequisites:
  • Java6+
  • Scala 2.10.x
  • Python 2.6 +
  • Spark 1.2.x
  • sbt ( In case of building Spark Source code)
  • GIT( If you use sbt tool)

Now we’ll see the installation steps :
  • Install Java 7 or later. Set JAVA_HOME and PATH variable as environment variables.
  • Download Scala 2.10 or Scala 2.11 and install. Set SCALA_HOME  and add %SCALA_HOME%\bin in   PATH variable in environment variables. To test whether Scala is installed or not, run following command
  • Next thing is Spark. Spark can be installed in two ways.
    •  Building Spark using SBT
    •  Use Prebuilt Spark package

 Building Spark with SBT :
  • Download SBT and install. Set SBT_HOME and PATH variable in environment variables.
  • Download source code from Spark website against any of the Hadoop version.
  • Run sbt assembly command to build the Spark package
  • You need to set Hadoop version also while building as follows : 
  •      sbt –Pyarn –pHadoop 2.3 assembly                                                                     

Using Spark Prebuilt Package:
  • Choose a Spark prebuilt package for Hadoop i.e.  Prebuilt for Hadoop 2.3/2.4 or later. Download and extract it to any drive i.e. D:\spark-1.2.1-bin-hadoop2.3
  • Set SPARK_HOME and add %SPARK_HOME%\bin in PATH in environment variables
  • Run following command on command line.                              
  • You’ll get and error for winutils.exe:
      Though we aren’t using Hadoop with Spark, but somewhere it checks for HADOOP_HOME variable in configuration. So to overcome this error, download winutils.exe and place it in any location (i.e. D:\winutils\bin\winutils.exe).

P.S. As per the Operating system version, this winutils.exe may vary. So in case, if it doesn't support to your OS, please find another one and use. You can refer this Problems running Hadoop on Windows link for winutils.exe.

  • Set HADOOP_HOME = D:\winutils in environment variable
  • Now, Re run the command “spark-shell’ , you’ll see the scala shell. For latest spark releases, if you get the permission error for /tmp/hive directory as given below:
  • The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rw-rw-rw- 
     You need to run following command :
D:\spark>D:\winutils\bin\winutils.exe chmod 777 D:\tmp\hive
  • For Spark UI : open http://localhost:4040/ in browser
  • For testing the successful setup you can run the example :   
  • It will execute the program and return the result :
Now your cluster is successfully launched, start writing your Java/Python/Scala programs…!!!! J

117 comments:

  1. Replies
    1. You should add one more option "\tmp\hive" without "D" for those who don't have "D"

      Delete
    2. That's the drive name, and people can easily understand, that they need to change it as per their system drive..

      Delete
  2. Replies
    1. Below link will work for Win7-32 bit OS :
      http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe

      Delete
  3. Thank you. It worked !!!

    ReplyDelete
  4. I followed the steps but finally I receive the error:
    C:\Spark\spark-1.4.0-bin-without-hadoop>spark-shell
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/
    FSDataInputStream
    at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSpa
    rkProperties$1.apply(SparkSubmitArguments.scala:111)
    at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSpa
    rkProperties$1.apply(SparkSubmitArguments.scala:111)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkPropert
    ies(SparkSubmitArguments.scala:111)
    at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArgume
    nts.scala:97)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:106)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStr
    eam
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 7 more

    Any idea what is wrong and how to solve this issue?

    ReplyDelete
    Replies
    1. Though you are using without-hadoop version, but It reads Hadoop enviroment variables while executing the shell. You can set HADOOP_HOME and path variable in conf/spark_env.cmd. Try it.

      Delete
    2. Hi - Followed the above steps but got the below error. I did try setting up HADOOP_HOME in path to C:\winutils\bin - and have setup HADOOP_HOME in conf\spark_env.cmd - export HADOOP_HOME="C:\winutils\bin"

      Can you please advice?

      Microsoft Windows [Version 6.1.7601]
      Copyright (c) 2009 Microsoft Corporation. All rights reserved.

      C:\pras\spark>scala -version
      Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL

      C:\pras\spark>bin\spark-shell
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
      at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply
      (SparkSubmitArguments.scala:117)
      at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply
      (SparkSubmitArguments.scala:117)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArgum
      ents.scala:117)
      at org.apache.spark.deploy.SparkSubmitArguments.(SparkSubmitArguments.scala:103)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:113)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
      at java.net.URLClassLoader$1.run(Unknown Source)
      at java.net.URLClassLoader$1.run(Unknown Source)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(Unknown Source)
      at java.lang.ClassLoader.loadClass(Unknown Source)
      at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
      at java.lang.ClassLoader.loadClass(Unknown Source)
      ... 7 more

      Delete
  5. Amazingly helpful! Thank you.

    ReplyDelete
  6. Thanks Nishu! It worked for me as well.

    ReplyDelete
  7. This comment has been removed by a blog administrator.

    ReplyDelete
  8. C:\spark-1.4.1-bin-hadoop2.6>bin\run-example CassandraCQLTest localhost 9160

    Why dont work me the example ? (Cassandra2.1.9 is running and the tables were created)

    ReplyDelete
    Replies
    1. the error is:

      Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
      at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)
      at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
      at scala.Option.getOrElse(Option.scala:120)
      at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1781)
      at org.apache.spark.rdd.RDD.count(RDD.scala:1099)
      at org.apache.spark.examples.CassandraCQLTest$.main(CassandraCQLTest.scala:110)
      at org.apache.spark.examples.CassandraCQLTest.main(CassandraCQLTest.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

      Delete
  9. Very helpful material to install Spark, I request please share a document in linux environment as well.

    ReplyDelete
    Replies
    1. Thanks Venu. Sure, will be sharing soon for linux environment.

      Delete
  10. C:\spark-1.5.0-bin-hadoop2.4\bin>spark-shell
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/laun
    cher/Main
    Caused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.Main
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    Could not find the main class: org.apache.spark.launcher.Main. Program will exi
    t.

    ReplyDelete
    Replies
    1. Have you followed all installation steps properly along with environment variables and sbt clean, assembly etc?

      Delete
  11. Hi....Can someone help me out in this

    ReplyDelete
  12. Thanks, this was much clearer and more straightforward than the circular links on the official Spark site.

    ReplyDelete
  13. Hi,
    I downloaded version 1.5 of Spark (spark-1.5.0-bin-hadoop2.4).
    I want to use Python. I installed Python 2.7.
    I had to install java 1.7.
     
    I add environment variables for python, java and spark.

    But it doesn't work.

    I have the following basic error:
    the specified path is not found.

    ReplyDelete
    Replies
    1. Your environment variable is not properly set it seems. Can you please check all the paths again?

      Delete
  14. Hi,

    Thanks for putting this together. I followed your steps using the pre-built spark installation. On trying to run spark-shell, I get the following error:

    15/09/24 17:02:22 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/bin/../lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar."
    15/09/24 17:02:22 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/bin/../lib/datanucleus-api-jdo-3.2.6.jar."
    15/09/24 17:02:22 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/bin/../lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Analytics-Tools/spark-1.5.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar."

    This seems to be related to the path settings. Any ideas?

    ReplyDelete
  15. Hi,

    I downloaded spark-1.5.1-bin-haddop2.6 and when I run spark-shell command, it returns with following error message -

    :10: error: not found: value sqlContext
    import sqlContext.sql

    Could anyone please help me with this!

    ReplyDelete
    Replies
    1. +1
      Downloaded the same version 1.5.1 and got the same issue. I still get the same error with or without the HADOOP_HOME set.

      Delete
  16. could anyone provide download link for winutils.exe for windows 32-bit.
    im getting below error when i used winutils.exe as mentioned in above explanation ,
    ocalhost): java.io.IOException: Cannot run program "D:\winutils\bin\winutils.exe
    ": CreateProcess error=216, This version of %1 is not compatible with the versio
    n of Windows you're running. Check your computer's system information to see whe
    ther you need a x86 (32-bit) or x64 (64-bit) version of the program, and then co
    ntact the software publisher

    ReplyDelete
  17. The latest (0.11.1) binary of Hadoop for Spark does not work under Windows. This seems to be because the Scala code tries to launch the script file named "mahout" (org.apache.mahout.sparkbindings findMahoutContextJars line:171) when building the Closeables list. Windows refuses to run a batch job that has no extension even if you change pathext or the registry. So you have to modify and rebuild the code to get it to run. If you do not have a required version of Visual Studio then you are out-of-luck.

    ReplyDelete
  18. This comment has been removed by the author.

    ReplyDelete
  19. I found the fixes to get it to run in Windows without rebuilding everything - such as if you do not have a recent version of MS-VS. (You will need a Win32 C++ compiler, but you can install MS VS Community Edition free.)

    see: http://stackoverflow.com/questions/25481325/how-to-set-up-spark-on-windows/33896022#33896022

    ReplyDelete
  20. I downloaded Apache spark release 1.5.2, prebuilt for Hadoop 2.4. I did set SPARK_HOME and HADOOP_HOME to point to winutils as per instructions.

    I am getting this error. Did I miss anything ?

    15/12/07 13:28:21 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
    15/12/07 13:28:22 WARN : Your hostname, PHXJ05429414 resolves to a loopback/non-reachable address: 10.68.232.112, but we couldn't find any external IP
    address!
    java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:171)
    at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:162)
    at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:160)
    at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:167)
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
    at $iwC$$iwC.(:9)
    at $iwC.(:18)
    at (:20)
    at .()
    at .(:7)
    at .()
    at $print()
    at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
    at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
    at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala
    :974)
    at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
    at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
    at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
    at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
    org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:559)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534)
    at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    ... 56 more

    :10: error: not found: value sqlContext
    import sqlContext.implicits._
    ^
    :10: error: not found: value sqlContext
    import sqlContext.sql

    ReplyDelete
    Replies
    1. Did you try setting SPARK_LOCAL_HOSTNAME to given IP address in config or commandline?

      Delete
  21. Does it matter, which way of installing Spark is used: 1) Building Spark using SBT OR
    Use Prebuilt Spark package 2) ?

    ReplyDelete
    Replies
    1. no, it doesn't matter. you can use whichever is easier for you

      Delete
  22. And what does this mean (if you choose 'Building Spark using SBT'): Set SBT_HOME and PATH variable in environment variables. I understand that SBT_HOME is the variable, but what should be put precisely into PATH?

    ReplyDelete
  23. To get a 100% working windows 7 product key, recommend you this site: www.motionkeys.com. Works absolutely perfectly!

    ReplyDelete
  24. For Windows 7 product key, any editions. Check this out: www.gankings.com. Works perfectly!!!

    ReplyDelete
  25. If you need product keys for windows 7, recommend you this page, you can click here to get one: www.facebook.com/Windows-7-Product-Key-1539410893045499. You're not gonna be disappointed. :)

    ReplyDelete
  26. If you guys wanna get a reliable Windows 7 product key, recommend you this page: plus.google.com/108152554871694437059/posts/izdExRxo4zN.

    ReplyDelete
  27. Thanks a lot for your post. I was facing difficulties in setting up the standalone Spark for Windows 10 64 bit machine. i was able to resolve it with your detailed explanation. Here is a link to my post where I shared my experience of configuring Spark on Windows 10
    http://www.nileshgule.com/2016/01/configure-standalone-spark-on-windows-10.html

    ReplyDelete
  28. Thanks for the solution installer spark on windows 7 ,I have a problem with the archive winutils.exe already that I have windows 7 32 bit . I would like know as I can charge my project of eclipse.the project have a connector between oracle and postgres. If I charge bin\spark-shell with the console open I can insert code of SCALA for I don`t know as charge my project on java.

    Thanks for your time!

    thanks for yours time!

    ReplyDelete
  29. This is where i got my Windows 7 product key, maybe you can find what you want here too: www.instructables.com/id/Windows-7-Product-Key.

    ReplyDelete
  30. Hi Nishu,

    I am new to Spark. I have completed the steps but I am getting the below error. Could you please assist here.

    /01/14 22:53:25 WARN : Your hostname, Paras-PC resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:8401%net9, but we couldn't find any external IP addr
    s!

    ReplyDelete
    Replies
    1. you need to set SPARK_LOCAL_HOSTNAME to localhost/127.0.0.1 or any IP address to resolve this.

      Delete
  31. Hi
    I am using a pre-built package spark-1.6.0-bin-hadoop2.6
    on Windows 7 with Java 7 and trying to launch using cygwin.
    I get Error: Could not find or load main class org.apache.spark.launcher.Main

    Tried quite a few suggestions on the web. Any specific pointers?
    Thank you

    ReplyDelete
  32. Hello i'm a new user to Spark. I did all steps as u said above. But when i wrote this cmd 'sbt -Pyran -Phadoop 2.3 assembly' , it says that it will take for a time to be availabe log. It took for about 5min and no changes. So, i close the command prompt and open new one and type again. Then it shows "java is not recognized as an external or internal command, operable program or batch file". Moreover i can't check my java version too. Before i do these i can check my java version. How do i do? Plz help me. Thanks

    ReplyDelete
    Replies
    1. You can setup it again from the first step and see, if you face the same issue again.

      Delete
  33. Many Thanks!! This works. Used winutils.exe from https://github.com/steveloughran/winutils/blob/master/hadoop-2.6.0/bin/winutils.exe

    ReplyDelete
  34. charm worker likened. thankyou.

    ReplyDelete
  35. sorry should have added got mine
    http://public-repo-1.hortonworks.com/hdp-win-alpha/winutils.exe

    ReplyDelete
  36. This comment has been removed by the author.

    ReplyDelete
  37. Really nice explanation ..Thanx a lot..

    ReplyDelete
  38. sprk-shell does not work permission issues come event after executing command winutils chmod 777 c:\tmp\hive

    ReplyDelete
    Replies
    1. install latest winutils.exe https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin
      run cmd as administrator for cjmod and spark-shell
      it works

      Delete
    2. Thanks for the update Mandos. Updated the winutil.exe url in post. :)

      Delete
  39. I have an error of failed to find spark assembly jar.
    What do I need to do in order for it to be assembled?

    ReplyDelete
  40. Thank you, the winutils error was causing the problem for me as well.

    ReplyDelete
    Replies
    1. Actually winutils.exe varies as per the operating system. As the one mentioned in post, worked for few users. We can always find the one which supports our operating system.

      Delete
  41. I've installed windows 7 on my laptop after buying license from ODosta Store, I want to activate it to get free upgrade from windows 7 professional to windows 10 home.
    I'm now confused, How to activate and upgrade to windows 10.
    Please mention within details.

    ReplyDelete
  42. hi Nishu,

    I really need your help. First of all thanks for this great tutorial. It worked for me.:)
    but now I am trying to run beeline but its is not working.

    C:\spark\spark-1.5.1-bin-hadoop2.4>bin\beeline
    Beeline version 1.5.1 by Apache Hive
    Exception in thread "main" java.lang.NoSuchMethodError: org.fusesource.jansi.internal.Kernel32.GetConsoleOutputCP()I
    at jline.WindowsTerminal.getConsoleOutputCodepage(WindowsTerminal.java:293)
    at jline.WindowsTerminal.getOutputEncoding(WindowsTerminal.java:186)
    at jline.console.ConsoleReader.(ConsoleReader.java:230)
    at jline.console.ConsoleReader.(ConsoleReader.java:221)
    at jline.console.ConsoleReader.(ConsoleReader.java:209)
    at org.apache.hive.beeline.BeeLine.getConsoleReader(BeeLine.java:834)
    at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:770)
    at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484)
    at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467)

    I have searched alot about this but I am not getting any help. Please help me with this. i really appreciate it. You are my last hope. Please

    ReplyDelete
  43. Thanks for sharing! Guys can also visit https://plus.google.com/103156713044838092165/posts/gque7ErHCVY515293381991299/ to get a genuine and cheap product key to activate your office 2010.

    ReplyDelete
  44. I've installed windows 8.1 on my laptop after buying license from ODosta Store, I want to activate it to get free upgrade from windows 7 professional to windows 10 home.
    I'm now confused, How to activate and upgrade to windows 10.
    Please mention within details.

    ReplyDelete
  45. Nice Blog , This is what I exactly Looking for , Keep sharing more blog .



    Spark Training in Chennai

    ReplyDelete
  46. Gracias por todas las llaves! Chicos si desea que las nuevas claves de productos genuinos de win 7 y que no quieren gastar mucho dinero para ello, a continuación, http://instalarclave.blogspot.kr será una buena opción para usted, precio de las claves 'es increíblemente bajo! Usted puede ir a echar un vistazo

    ReplyDelete
  47. Thx ! that's really helped me

    ReplyDelete
  48. Simple & Superb.. Much Informative..Thanks for sharing :)

    ReplyDelete
  49. Thank you for the good information. We also provide Apache online training.Go here if you’re looking for information on Apache Training.

    ReplyDelete


  50. I've installed windows 7 on my laptop after buying license from cheaproduct, I want to activate it to get free upgrade from windows 7 professional to windows 10 home.
    I'm now confused, How to activate and upgrade to windows 10.
    Please mention within details.


    ReplyDelete
  51. On typing sbt assembly to cmd it is giving following error:
    java HotSpot Client VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
    [info] set current project to mnc
    [error] Not a valid command: assembly
    [error] Not a valid project ID: assembly
    [error] Not a valid configuration: assembly
    [error] Not a valid key: assembly
    [error] assembly
    [error]
    please help me out of this error.

    ReplyDelete
  52. Great Tutorial, Thanks.

    ReplyDelete
  53. The instructions in this blog is clear and it worked for me, wicked!!!!

    ReplyDelete
  54. you can visit to the below url , they are offering very good videos on Hadoop:

    For free videos from previous sessions refer:
    http://hadoopbigdatatutorial.com/hadoop-training/big-data-tutorial

    ReplyDelete
  55. You will get a good guidance on installation of php, apache and mysql here
    http://prodevelopertutorial.blogspot.in/2016/08/how-to-install-apache-server-mysql-and-php-on-window.html

    ReplyDelete
  56. thanks a lot..

    ReplyDelete
  57. Hi Nishu,
    I have installed using the second method and getting the warning -
    NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

    ReplyDelete
  58. Share with you a good site that you can get cheap product keys from there: www.vanskeys.com, all versions of windows keys and office keys can be found in that site.

    ReplyDelete
  59. Windows 7 support is over, but there's no doubt, As Windows 7 is the most recommended OS, Which through user can do almost all work without any problem, Where I recommend you to activate your OS being purchased its license code from ODosta Store
    After OS activation, You can easily get windows 10 remote tech support to run all old programs and get windows 10 updates.
    You can upgrade your OS to windows 10, But you can face some technical issues, So I recommend you to have clean installation of windows 10 and activate it using legal license.

    ReplyDelete
  60. hi welcome to this blog.it will be really helpful to many peoples.thank you for sharing this blog.
    apachespark training

    ReplyDelete
  61. Siddharth Sahani31 August 2016 at 14:39

    It was the most convenient installation, with your Blog. Thanks a tonne, Nishu.

    ReplyDelete
  62. Great! article, It's working for me. Can you also share next steps for writing Java code in 1.8, intellij. Any simple Hello world programming steps from here will be very helpful

    ReplyDelete
  63. https://plus.google.com/111985418905952341154/posts/CPp9BGNDPAT

    ReplyDelete
  64. It works well. Thanks for sharing article. When i execute the command - run-example JavaSparkPi , it works well. But i execute the program - run-example JavaSparkSQL , it shows error.

    How to rectify


    c:\Spark\bin>run-example JavaSparkSQL
    java.lang.ClassNotFoundException: org.apache.spark.examples.JavaSparkSQL
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:538)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

    c:\Spark\bin>

    ReplyDelete
  65. Hi ... I am getting the below error while starting the shell command.

    >spark-shell.cmd
    'Spark\spark-2.0.1-bin-hadoop2.7\spark-2.0.1-bin-hadoop2.7\bin\..\jars""\' is no
    t recognized as an internal or external command,
    operable program or batch file.
    Failed to find Spark jars directory.
    You need to build Spark before running this program.

    ReplyDelete
    Replies
    1. happens if the pathname of the spark folder has a space, for example c:\progfram files\spark. Install somewhere like c:\spark

      Delete
  66. I use on ubuntu linux 64 with spark 2.0.1-bin-hadoop-2.7,maven 3.3.9, mahout 3.3.9 and mahout 0.12.2.When I type bin/mahout spark-shell,the following error is appeared:
    cm@ubuntu:/usr/local/apache-mahout-distribution-0.12.2$ bin/mahout spark-shell
    MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath.
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/repl/SparkILoop
    at org.apache.mahout.sparkbindings.shell.Main.main(Main.scala)
    Caused by: java.lang.ClassNotFoundException: org.apache.spark.repl.SparkILoop
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 1 more
    plz advice me,how can I solve?

    ReplyDelete
    Replies
    1. Did you check if you spark-repl has SparkILoop.scala class? You can refer https://github.com/apache/spark/blob/master/repl/scala-2.10/src/main/scala/org/apache/spark/repl/SparkILoop.scala link to traverse the class path. Seems installation didn't happen properly or any classpath is missing. Can you check once?

      Delete
  67. Thanks.
    That was an awesome blog. Fantastic.
    Hepled me a lot after continuous struggle.

    ReplyDelete
  68. i have followed all, the steps but getting the msg " file name or directory name is not correct" when i write the command "C:\spark-2.0.0-bin-without-hadoop\bin\spark-shell" plz reply as I have set java hme, scala home and spark home

    ReplyDelete
  69. This was neat. Kudos for making this..

    ReplyDelete
  70. Hehe....i was following this blog to install spark...later realized its yours....Too good.

    ReplyDelete
  71. You saved my day. Extremely helpful. Thanks for the great post.

    ReplyDelete
  72. Whenever i enter the command spark-shell in cmd it gives me an Error saying

    "The filename, directory name, or volume label syntax is incorrect."

    Please Help!

    ReplyDelete
  73. Getting this error

    java.io.IOException: Failed to delete: C:\Users\vishnu.s\AppData\Local\Temp\spark-c1c7080c-1726-4c76-aecb-b9230f3e7df0\userFiles-fd6c46d1-f5b5-4bca-bc20-893713
    at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)
    at org.apache.spark.SparkEnv.stop(SparkEnv.scala:102)
    at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1842)
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1841)
    at org.apache.spark.sql.SparkSession.stop(SparkSession.scala:665)

    ReplyDelete
  74. Nice, contact at Gmail Contact Number Canada1-888-414-1732, our expert will solve your problem.

    ReplyDelete
  75. Nice blog its very useful information you given thanks for sharing.
    ETL Testing Online Training

    ReplyDelete
  76. Thank you so much buddy... It helped a lot in setting up the env. :)

    ReplyDelete