Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ZEPPELIN-3840] Zeppelin on Kubernetes #3240

Closed
wants to merge 39 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
d2f3d5b
add k8s-standard launcher module
Leemoonsoo Nov 5, 2018
07489f7
kubectl with exec
Leemoonsoo Nov 18, 2018
5f602a6
K8sRemoteInterpreterProcess
Leemoonsoo Nov 20, 2018
52bb6c7
add k8s dir in package
Leemoonsoo Nov 21, 2018
36cf391
correct plugin name
Leemoonsoo Nov 21, 2018
58f9f19
add rbac
Leemoonsoo Nov 21, 2018
2fd2ac8
kubernetes mode configuration
Leemoonsoo Nov 21, 2018
9f1b7a1
run kubernetes launcher
Leemoonsoo Nov 21, 2018
0dea383
create and connect interpreter pod
Leemoonsoo Nov 22, 2018
18b8f68
print spec file contents on debug log
Leemoonsoo Nov 22, 2018
86e8764
add services on RBAC
Leemoonsoo Nov 22, 2018
7fe9823
interpreter pod cascade delete on zeppelin-server delete
Leemoonsoo Nov 22, 2018
263d859
use headless service for interpreter pod
Leemoonsoo Nov 22, 2018
7a87367
configure spark on kubernetes
Leemoonsoo Nov 22, 2018
a4072e6
add signal handler
Leemoonsoo Nov 22, 2018
2960dcb
configure namespace
Leemoonsoo Nov 22, 2018
b0e2c36
Rbac role, rolebinding
Leemoonsoo Nov 22, 2018
0d472ea
load properties and environment variables
Leemoonsoo Nov 23, 2018
f4166ad
make spark container image configurable
Leemoonsoo Nov 23, 2018
2b579ff
let user override namespace
Leemoonsoo Nov 23, 2018
f305611
rename file
Leemoonsoo Nov 23, 2018
0f7c0d4
add license
Leemoonsoo Nov 23, 2018
9341fcb
install kubectl and configure log4j in docker image
Leemoonsoo Nov 23, 2018
3078bac
spark ui support
Leemoonsoo Nov 24, 2018
ec09b8b
add test
Leemoonsoo Nov 24, 2018
e9ce64f
update dockerfile
Leemoonsoo Nov 24, 2018
64a56b5
Add docs
Leemoonsoo Nov 24, 2018
b4f58a9
sig term for quick termination
Leemoonsoo Nov 24, 2018
fa36c18
Make spark master configurable
Leemoonsoo Nov 24, 2018
e2dc88a
suppress error log when wait target is already removed
Leemoonsoo Nov 24, 2018
1a3a070
Set ownerreference to Role and Rolebinding of interpreter
Leemoonsoo Nov 24, 2018
3b652a4
Make spark executor set ownerreference correctly
Leemoonsoo Nov 24, 2018
11960dd
update corresponding test as well
Leemoonsoo Nov 24, 2018
22e090f
logger -> LOGGER
Leemoonsoo Nov 25, 2018
9ab6fc4
address code review
Leemoonsoo Nov 27, 2018
993a0e4
document configurations
Leemoonsoo Nov 27, 2018
4e7d817
localtest.me -> local.zeppelin-project.org
Leemoonsoo Nov 28, 2018
423412a
zeppelin.k8s.mode -> zeppelin.run.mode
Leemoonsoo Nov 30, 2018
0100a36
update how it works on docs, add some comments on yaml files
Leemoonsoo Dec 7, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
interpreter pod cascade delete on zeppelin-server delete
  • Loading branch information
Leemoonsoo committed Nov 22, 2018
commit 7fe9823b1f2d6f311104af38b674b77494e2e970
18 changes: 18 additions & 0 deletions k8s/interpreter/100-interpreter-pod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ metadata:
app: {{POD_NAME}}
interpreterGroupId: {{INTP_ID}}
interpreterSettingName: {{INTP_SETTING}}
{% if OWNER_UID is defined and OWNER_UID %}
ownerReferences:
- apiVersion: v1
controller: false
blockOwnerDeletion: false
kind: Pod
name: {{OWNER_NAME}}
uid: {{OWNER_UID}}
{% endif %}
spec:
automountServiceAccountToken: false
restartPolicy: Never
Expand All @@ -24,6 +33,15 @@ apiVersion: v1
metadata:
namespace: {{NAMESPACE}}
name: {{POD_NAME}} # keep Service name the same to Pod name.
{% if OWNER_UID is defined and OWNER_UID %}
ownerReferences:
- apiVersion: v1
controller: false
blockOwnerDeletion: false
kind: Pod
name: {{OWNER_NAME}}
uid: {{OWNER_UID}}
{% endif %}
spec:
ports:
- name: intp
Expand Down
10 changes: 10 additions & 0 deletions k8s/zeppelin.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,16 @@ spec:
value: /zeppelin
- name: ZEPPELIN_SERVER_RPC_PORTRANGE
value: 12320:12320
- name: POD_UID
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.uid
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
---
kind: Service
apiVersion: v1
Expand Down
2 changes: 1 addition & 1 deletion zeppelin-plugins/launcher/k8s-standard/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
<dependency>
<groupId>com.hubspot.jinjava</groupId>
<artifactId>jinjava</artifactId>
<version>2.0.11-java7</version>
<version>2.4.12</version>
</dependency>
</dependencies>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -248,10 +248,24 @@ Map<String, Object> getTemplateBindings() throws IOException {
var.put("CALLBACK_PORT", zeppelinServiceRpcPort); // interpreter.sh -p
var.put("INTP_SETTING", interpreterSettingName); // interpreter.sh -g
var.put("INTP_REPO", "/tmp/local-repo"); // interpreter.sh -l
var.put("OWNER_UID", ownerUID());
var.put("OWNER_NAME", ownerName());
var.putAll(Maps.fromProperties(properties)); // interpreter properties override template variables
return var;
}

/**
* Get UID of owner (zeppelin-server pod) for garbage collection
* https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/
*/
private String ownerUID() {
return System.getenv("POD_UID");
}

private String ownerName() {
return System.getenv("POD_NAME");
}

private String getRandomString(int length) {
char[] chars = "abcdefghijklmnopqrstuvwxyz".toCharArray();

Expand Down