Skip to content

Commit

Permalink
[ZEPPELIN-1192] Block pyspark paragraph hang.
Browse files Browse the repository at this point in the history
### What is this PR for?
This PR block pyspark paragraph hang.

### What type of PR is it?
Bug Fix

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1192

### How should this be tested?
1. build zeppelin without pyspark (just mvn clean package -DskipTests)
2. open note and set paragraph interpreter as "spark.pyspark"
3. hit "Ctrl+." for auto completion.
4. try run paragraph.
- please refer to the screenshot.

### Screenshots (if appropriate)
- before
![b](https://cloud.githubusercontent.com/assets/3348133/16881827/ee30e248-4af6-11e6-9409-7e7b9f622121.gif)

- after
![a](https://cloud.githubusercontent.com/assets/3348133/16881840/f4d6d2ec-4af6-11e6-89b3-1e4e2806a742.gif)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no

Author: astroshim <[email protected]>

Closes apache#1193 from astroshim/ZEPPELIN-1192 and squashes the following commits:

9953b1a [astroshim] Merge branch 'master' into ZEPPELIN-1192
4b26ab3 [astroshim] replace hardcoded value.
670cbc8 [astroshim] log timing out.
e95f819 [astroshim] Merge branch 'master' into ZEPPELIN-1192
dbd649c [astroshim] Merge branch 'master' of https://github.com/astroshim/zeppelin into ZEPPELIN-1192
6eb1666 [astroshim] timeout value defines as a constant.
288eca7 [astroshim] block infinite loop.
  • Loading branch information
astroshim authored and bzz committed Aug 17, 2016
1 parent 47b1931 commit 051929d
Showing 1 changed file with 10 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ public class PySparkInterpreter extends Interpreter implements ExecuteResultHand
private ByteArrayOutputStream input;
private String scriptPath;
boolean pythonscriptRunning = false;
private static final int MAX_TIMEOUT_SEC = 10;

public PySparkInterpreter(Properties property) {
super(property);
Expand Down Expand Up @@ -316,7 +317,7 @@ public InterpreterResult interpret(String st, InterpreterContext context) {
long startTime = System.currentTimeMillis();
while (pythonScriptInitialized == false
&& pythonscriptRunning
&& System.currentTimeMillis() - startTime < 10 * 1000) {
&& System.currentTimeMillis() - startTime < MAX_TIMEOUT_SEC * 1000) {
try {
pythonScriptInitializeNotifier.wait(1000);
} catch (InterruptedException e) {
Expand Down Expand Up @@ -423,8 +424,15 @@ public List<InterpreterCompletion> completion(String buf, int cursor) {
}

synchronized (statementFinishedNotifier) {
while (statementOutput == null) {
long startTime = System.currentTimeMillis();
while (statementOutput == null
&& pythonScriptInitialized == false
&& pythonscriptRunning) {
try {
if (System.currentTimeMillis() - startTime < MAX_TIMEOUT_SEC * 1000) {
logger.error("pyspark completion didn't have response for {}sec.", MAX_TIMEOUT_SEC);
break;
}
statementFinishedNotifier.wait(1000);
} catch (InterruptedException e) {
// not working
Expand Down

0 comments on commit 051929d

Please sign in to comment.