How to integrate JupyterHub with the existing Cloudera cluster

[mdivk]

Hello,

We have an existing Cloudera cluster, python 3.3, pyspark is running python 2.7.14.

We are able to set up Jupyterhub on the same host with both python 2 and 3

How can we set up the environment in JupyterHub so that we can create Spark Context (python 2/spark 1.6) or Spark Session (python 3/spark 2) and access the HDFS data on the Cloudera Cluster?

Example would be greatly appreciated.

Thank you in advance.

BTW: the authentication part is very nice.

[mdivk]

The following is what I did and where I stuck at:
create a start_jupyterhun.sh under .jupyter folder

export SPARK_HOME=/opt/cloudera/parcels/SPARK2
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
sudo jupyterhub

Starting the Hub successfully, starting a new notebook,
first cell:

from pyspark import SparkContext, SparkConf
still getting error below:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9a0cac67cda7> in <module>()
----> 1 from pyspark import SparkContext, SparkConf

ImportError: No module named 'pyspark'

any missing here?

I understand Livy+SparkMagic might be the better way but at the moment it is just about accessing pyspark module which already exists in the host, my understanding is this could be just some environment treak. I could be wrong though but I do want to get this fixed first and then try out other option like Livy+SparkMagic or cdsw

Thank you very much, any input/hint is appreciated.

[SivaAccionLabs]

"env": {
  "HADOOP_CONF_DIR":"{{ hadoop_conf_dir }}",
  "PYSPARK_PYTHON":"{{ anaconda_home }}/bin/python",
  "SPARK_HOME": "{{ wrapper_spark_home }}",
  "WRAPPED_SPARK_HOME": "{{ spark_home }}",
  "PYTHONPATH": "{{ app_packages_home }}/lib/python2.7/site-packages:{{ jupyter_extension_venv }}/lib/python2.7/site-packages:{{ spark_home }}/python:{{ spark_home }}/python/lib/py4j-0.9-src.zip",
  "PYTHONSTARTUP": "{{ spark_home }}/python/pyspark/shell.py",
  "PYSPARK_SUBMIT_ARGS": "--master yarn-client --jars {{ spark_home }}/lib/spark-examples.jar pyspark-shell"
 }

[mdivk]

Thank you SivaMaplelabs,

This is the first time working with JupyterHub or Jupyter, it would be greatly appreciated if you can be more specific and provide more detail on the snippet in your reply, looks like it is from somewhere within some configuration file? where do I put your code to?

Thank you in advance.

[mdivk]

I found this in my host:

/opt/cloudera/parcels/Anaconda/share/jupyter/kernels/python2

[hadoopuser@hadoop02 python2]$ cat kernel.json
{
 "display_name": "Python 2",
 "language": "python",
 "argv": [
  "/opt/cloudera/parcels/Anaconda-5.0.1/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ]

is this where your code snippet from? but I don't see "env" here.

[SivaAccionLabs]

@mdivk Have a look at the following GitHub link. We are deploying jupyter/jupyterhub on HDP and CDH with salt.
https://github.com/pndaproject/platform-salt/tree/develop/salt/jupyter

[mdivk]

hmmm, thank you Siva @SivaMaplelabs, but I only want to have the env be fixed for now.

[SivaAccionLabs]

@mdivk following is working config for Pyspark/Pyspark2 kernel.
Pyspark

{
 "display_name": "PySpark/Python2 (Anaconda)",
 "language": "python",
 "argv": [
  "{{ anaconda_home }}/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "env": {
  "HADOOP_CONF_DIR":"{{ hadoop_conf_dir }}",
  "PYSPARK_PYTHON":"{{ anaconda_home }}/bin/python",
  "SPARK_HOME": "{{ wrapper_spark_home }}",
  "WRAPPED_SPARK_HOME": "{{ spark_home }}",
  "PYTHONPATH": "{{ app_packages_home }}/lib/python2.7/site-packages:{{ jupyter_extension_venv }}/lib/python2.7/site-packages:{{ spark_home }}/python:{{ spark_home }}/python/lib/py4j-0.9-src.zip",
  "PYTHONSTARTUP": "{{ spark_home }}/python/pyspark/shell.py",
  "PYSPARK_SUBMIT_ARGS": "--master yarn-client --jars {{ spark_home }}/lib/spark-examples.jar pyspark-shell"
 }
}

Pyspark2

{
 "display_name": "PySpark2/Python2 (Anaconda)",
 "language": "python",
 "argv": [
  "{{ anaconda_home }}/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "env": {
  "HADOOP_CONF_DIR":"{{ hadoop_conf_dir }}",
  "PYSPARK_PYTHON":"{{ anaconda_home }}/bin/python",
  "SPARK_MAJOR_VERSION":"2",
  "SPARK_HOME": "{{ wrapper_spark_home }}",
  "WRAPPED_SPARK_HOME": "{{ spark2_home }}",
  "PYTHONPATH": "{{ app_packages_home }}/lib/python2.7/site-packages:{{ jupyter_extension_venv }}/lib/python2.7/site-packages:{{ spark2_home }}/python:{{ spark2_home }}/python/lib/py4j.zip",
  "PYTHONSTARTUP": "{{ spark2_home }}/python/pyspark/shell.py",
  "PYSPARK_SUBMIT_ARGS": "--master yarn-client --jars {{ spark2_home }}/examples/jars/spark2_examples.jar pyspark-shell"
 }
}

[SivaAccionLabs]

You can replace the HADOOP_CONF_DIR, PYSPARK_PYTHON and other env variables paths as per your environment.

[mdivk]

Thank you Siva @SivaMaplelabs:

1. is the kernel config the one from this location: /opt/cloudera/parcels/Anaconda/share/jupyter/kernels/python2?

2. can I simply replace the the entire file with your sample? any extra step(s) like restart the service or reboot do I need to go through?

Thanks.

[mdivk]

You can replace the HADOOP_CONF_DIR, PYSPARK_PYTHON and other env variables paths as per your environment.

Both are empty in my current cluster.
Some environment here:

hadoop: /etc/hadoop
$HADOOP_HOME: empty
spark: /etc/spark
spark2: /etc/spark2
$SPARK_HOME: empty
$SPARK2_HOME: empty

which pyspark
/usr/bin/pyspark
which pyspark2
/usr/bin/pyspark2

ls /etc/hadoop
conf  conf.cloudera.hdfs  conf.cloudera.yarn
ls /etc/spark
conf  conf.cloudera.spark_on_yarn
ls /etc/spark2
conf  conf.cloudera.spark2_on_yarn

Thank you again for your patience.

[mdivk]

This is what I tried out and failed again:
/opt/cloudera/parcels/Anaconda/share/jupyter/kernels/python2/kernel.json:

{
 "display_name": "PySpark/Python2 (Anaconda)",
 "language": "python",
 "argv": [
  "{{ anaconda_home }}/bin/python",
  "-m",
  "ipykernel",
  "-f",
  "{connection_file}"
 ],
 "env": {
  "HADOOP_CONF_DIR":"/etc/hadoop/conf",
  "PYSPARK_PYTHON":"/opt/cloudera/parcels/Anaconda/bin/python",
  "SPARK_HOME": "/etc/spark",
  "WRAPPED_SPARK_HOME": "/etc/spark",
  "PYTHONPATH": "/opt/cloudera/parcels/CDH/lib/spark/python/:/opt/cloudera/parcels/CDH/lib/spark/python/lib/py4j*src.zip",
  "PYTHONSTARTUP": "/opt/cloudera/parcels/CDH/lib/spark/python/pyspark/shell.py",
  "PYSPARK_SUBMIT_ARGS": "--master yarn-client" 
 }
}

The JupyterHub is started with sudo jupyterhub in the folder of ~/.jupyter
Once started, open a new notebook and run the code in the first cell:

from pyspark import SparkContext, SparkConf

Failed out as below:

ImportError Traceback (most recent call last)
in ()
----> 1 from pyspark import SparkContext, SparkConf

ImportError: No module named 'pyspark'

I am wondering if I started the wrong kernel? The dashboard shows "Python 3" as the kernel.
Where is the kernel folder for JupyterHub (not for Jupyter)?

I don't see a kernel.json file under the folder of /usr/local/share/jupyter/hub

[hadoopuser@hadoop02 .jupyter]$ ls -la /usr/local/share/jupyter/hub
total 16
drwxr-xr-x 4 root root 4096 Aug 23 11:04 .
drwxr-xr-x 4 root root 4096 Aug 23 11:04 ..
drwxr-xr-x 7 root root 4096 Aug 23 11:04 static
drwxr-xr-x 2 root root 4096 Aug 23 11:04 templates
[hadoopuser@hadoop02 .jupyter]$ find /usr/local/share/jupyter/hub -name *.json
/usr/local/share/jupyter/hub/static/components/font-awesome/.bower.json
/usr/local/share/jupyter/hub/static/components/font-awesome/bower.json
/usr/local/share/jupyter/hub/static/components/font-awesome/composer.json
/usr/local/share/jupyter/hub/static/components/font-awesome/component.json
/usr/local/share/jupyter/hub/static/components/bootstrap/.bower.json
/usr/local/share/jupyter/hub/static/components/bootstrap/bower.json
/usr/local/share/jupyter/hub/static/components/jquery/.bower.json
/usr/local/share/jupyter/hub/static/components/jquery/package.json
/usr/local/share/jupyter/hub/static/components/jquery/bower.json
/usr/local/share/jupyter/hub/static/components/jquery/composer.json
/usr/local/share/jupyter/hub/static/components/jquery/component.json
/usr/local/share/jupyter/hub/static/components/moment/.bower.json
/usr/local/share/jupyter/hub/static/components/moment/bower.json
/usr/local/share/jupyter/hub/static/components/requirejs/.bower.json
/usr/local/share/jupyter/hub/static/components/requirejs/bower.json
[hadoopuser@hadoop02 .jupyter]$

[SivaAccionLabs]

@mdivk Is SPARK_HOME correct in your kernel.json?
In general, SPARK_HOME path would be

spark_home = '/opt/cloudera/parcels/CDH/lib/spark' in CDH
spark_home = '/usr/hdp/current/spark-client' in HDP

Also update your PYSPARK_SUBMIT_ARGS as bellow

--master yarn-client --jars {{ spark_home }}/lib/spark-examples.jar pyspark-shell

[mdivk]

Thank you Siva.
It's becoming a lengthy thread now, with many attempts to fix the issue, it is still there, I am to close this one and open a new thread to make it clearer:
#2133

[SivaAccionLabs]

ok