-
Notifications
You must be signed in to change notification settings - Fork 223
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support dynamic load new kernel spec #1127
Comments
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗 |
Hi @wuyueandrew - it is not typically necessary to restart EG (or any jupyter server) to use additional kernelspecs added after the server has started. When kernelspecs are requested via, for example, However, if this is a Kubernetes deployment, we are using the If this is not a containerized env, then you could get the dynamic behavior you need by either not setting If you're using a containerized environment, then, yes, we should introduce a way to NOT set the whitelist, so that users can have the ability to add kernelspecs dynamically. Since the configuration file would live in the container, accessing would require mounts and an adjustment to the If K8s, perhaps we could allow for the entry in values.yaml to be:
which will then set How often are you creating kernelspecs? (Just curious) |
thanks kevin, u answer is helpful, i'll try ur solution later. Typically, my colleagues design kernel spec(python env and some wheel) for a custome scenario and most of default kernels can cover scenario. The frenquence of creating kernelspecs depends on our customers.So far, we don't create kernelspecs often. |
Hi, kevin, thanks again. After try ur solution, it cannot solve. I used helm to deployment eg on k8s. So maybe i misunderstood. Is kernel.whitelist=* is not support yet , and the only way of solving this is not deploy on k8s. |
Correct. Support to recognize
Because we're setting the CLI options in the
Using helm to redeploy EG with a new kernelspec will also not work because it restarts EG. Also note that if we were to enable support for dask_python_yarn_remote, spark_python_kubernetes, python_distributed, spark_python_operator, python_docker, spark_python_yarn_client, python_kubernetes, spark_python_yarn_cluster, python_tf_docker, spark_R_conductor_cluster, python_tf_gpu_docker, spark_R_kubernetes, python_tf_gpu_kubernetes, spark_R_yarn_client, python_tf_kubernetes, spark_R_yarn_cluster, R_docker, spark_scala_conductor_cluster, R_kubernetes, spark_scala_kubernetes, scala_docker, spark_scala_yarn_client, scala_kubernetes, spark_scala_yarn_cluster, spark_python_conductor_cluster As a result, you'd probably want to remove a majority of these since they won't apply to your environment. So, I see there being two ways forward to support dynamic kernelspecs on K8s:
In both cases, you would NOT use helm to redeploy EG in order to add a kernelspec. You would merely update the appropriate location(s) with the new kernelspec. I think option 2 would be more difficult on an operator/admin than option 1, so would lean toward implementing option 1. I can look into this later today or tomorrow if that option would work for you. |
Apprecicate for u patient answer , very kind. |
Hi @wuyueandrew. Please see #1131. The approach I took is for the operator to remove or comment out the |
Thank you. I'm look forward to V3.0 |
Problem
for now, we should restart eg after install a new kernel spec and it is not convenient. cuz other users cannot connect to gateway
due to restarting eg.
Proposed Solution
only support installing new kernel specs. elder kernel in used are not support to escape running problem.
The text was updated successfully, but these errors were encountered: