Livy:337 Binding RPCServer to user provided port and not random port #334

pralabhkumar · 2017-05-18T12:08:14Z

#Livy:337 : Bind RPCServer to user provided port https://issues.cloudera.org/browse/LIVY-337

codecov-io · 2017-05-21T06:56:50Z

Codecov Report

Merging #334 into master will increase coverage by 0.17%.
The diff coverage is 89.47%.

@@             Coverage Diff              @@
##             master     #334      +/-   ##
============================================
+ Coverage     70.46%   70.63%   +0.17%     
- Complexity      726      730       +4     
============================================
  Files            96       96              
  Lines          5123     5153      +30     
  Branches        774      776       +2     
============================================
+ Hits           3610     3640      +30     
  Misses          996      996              
  Partials        517      517

Impacted Files	Coverage Δ	Complexity Δ
...c/src/main/java/com/cloudera/livy/rsc/RSCConf.java	`87.75% <100%> (+0.12%)`	`7 <0> (ø)`	⬇️
...main/java/com/cloudera/livy/rsc/rpc/RpcServer.java	`86.77% <89.18%> (+0.9%)`	`13 <3> (+2)`	⬆️
...n/java/com/cloudera/livy/rsc/driver/RSCDriver.java	`77.58% <0%> (+1.29%)`	`41% <0%> (+2%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2abb8a3...3f58233. Read the comment docs.

pralabhkumar · 2017-05-21T10:08:23Z

@zjffdu @alex-the-man :
Hi I have created the pull request for the initial version for Jira https://issues.cloudera.org/browse/LIVY-337

Build : 238d238
I am still working on this . Could you please review code/approach to see if its in right direction

pralabhkumar · 2017-05-22T05:59:43Z

@zjffdu @alex-the-man : Could you please review the pull request. Thanks

pralabhkumar · 2017-05-24T02:23:44Z

@zjffdu @alex-the-man : Please let me know , if there any active development going in livy . Or should I close this pull request . I am using Livy in my organization and needed these changes .

pralabhkumar · 2017-05-26T11:10:02Z

@zjffdu : Any Update ?

zjffdu · 2017-05-26T22:39:09Z

@pralabhkumar sorry for late response. It seems user need to specify 2 properties for the port which is a little complicated to me. Could we just use one property ? e.g. launcher.port.range=5000-5500

pralabhkumar · 2017-05-28T06:53:55Z

@zjffdu Thanks for the suggestion , working on it . Will push the new changes in couple of days

pralabhkumar · 2017-05-30T02:03:23Z

@zjffdu . Have done the changes as suggested .

launcher.port.range is the parameter which user should set . This parameter is the only parameter user should set.

launcher.port is there in the RSCConf but there is no effect of user setting it . This property is now used to communicate launcher port number from ContextLauncer to RSCDriver.

Please review the changes .

zjffdu · 2017-05-31T05:36:19Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

@@ -61,18 +63,74 @@
  private static final SecureRandom RND = new SecureRandom();

  private final String address;
-  private final Channel channel;
+  private  Channel channel;


zjffdu · 2017-05-31T05:36:58Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

  public RpcServer(RSCConf lconf) throws IOException, InterruptedException {
    this.config = lconf;
+    this.portRange=config.get(LAUNCHER_PORT_RANGE);


miss space around =

zjffdu · 2017-05-31T05:38:19Z

It is weird that I found some code style issue, but CI still pass. @jerryshao Could you help check whether it is caused by the maven project structure ?

jerryshao · 2017-05-31T05:40:49Z

Ok, let me check it. Maybe Java code style check is not worked well.

zjffdu · 2017-05-31T05:41:40Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

+      }catch(BindException e){
+        LOG.warn("RPC not able to connect port "+ startingPortNumber);
+      }
+    }


What would happen when there is no available port in this range

Have Changed the code , to handle the case if no available port . It will throws exception and fails gracefully.

Please let me know , if that ok .

@zjffdu Working on this .

@zjffdu done. Graceful failure if not port available (as would have happened if the random port is not available)

pralabhkumar · 2017-05-31T14:40:02Z

@zjffdu
@jerryshao

I have done the changes as suggested by @zjffdu . But travis build doesn't run (don't know why) . Please review the changes .

pralabhkumar · 2017-06-01T06:52:42Z

@zjffdu
@jerryshao

Facing following issue while building it in Travis . Though I have not done any changes with respect to below error

ImportError: No module named six

----------------------------------------

Command "/opt/python/2.7.12/bin/python2.7 -u -c "import setuptools, tokenize;file='/tmp/pip-build-kE1PPR/configparser/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-LNZmV5-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-kE1PPR/configparser/

The command "pip install --user requests pytest flaky flake8 requests-kerberos" failed and exited with 1 during .

pralabhkumar · 2017-06-01T10:32:26Z

@zjffdu
@jerryshao

Please help me with above issue , i am sure it has nothing to do with my part of code .

jerryshao · 2017-06-01T12:35:33Z

Looks like Travis is broken, let me try to fix it.

pralabhkumar · 2017-06-01T15:07:45Z

@jerryshao
Thanks for fixing it above issue . But Still there are some issues ,which may not be related my code changes . Please help

1 There is no port available to launch RPCServer . I used the same port which used for this build
4a7e219 . But now , its saying no port available in the range .

2 Some of the builds are giving following error . Please help

basic interactive session *** FAILED *** (5 minutes, 3 seconds)
The code passed to eventually never returned normally. Attempted 305 times over 5.012684563933334 minutes. Last failure message: assertion failed: Session 0 state dead doesn't equal one of Set(idle). (LivyRestClient.scala:100)
- Final session state: SessionSnapshot(0,Some(application_1496328962772_0001),dead,AppInfo(None,Some(http://testing-gce-05fca65b-482d-4c3e-8cf6-39a9f86cb83c:36897/cluster/app/application_1496328962772_0001)),Vector( at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302), at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82), at java.util.concurrent.FutureTask.run(FutureTask.java:262), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), at java.lang.Thread.run(Thread.java:745), , , Container exited with a non-zero exit code 15, Failing this attempt. Failing the application.))
- YARN diagnostics: Application application_1496328962772_0001 failed 1 times due to AM Container for appattempt_1496328962772_0001_000001 exited with exitCode: 15
  For more detailed output, check application tracking page:http://testing-gce-05fca65b-482d-4c3e-8cf6-39a9f86cb83c:36897/cluster/app/application_1496328962772_0001Then, click on links to logs of each attempt.
  Diagnostics: Exception from container-launch.
  Container id: container_1496328962772_0001_01_000001
  Exit code: 15
  Stack trace: ExitCodeException exitCode=15:
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
  at org.apache.hadoop.util.Shell.run(Shell.java:479)
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
  at java.util.concurrent.FutureTask.run(FutureTask.java:262)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:745)
  Container exited with a non-zero exit code 15
  Failing this attempt. Failing the application.

pralabhkumar · 2017-06-01T15:10:37Z

HI jerryshao <https://github.com/jerryshao> Thanks for fixing it above issue . But Still there are some issues ,which may not be related my code changes . Please help 1 There is no port available to launch RPCServer . I used the same port which used for this build 4a7e219 <4a7e219> . But now , its saying no port available in the range . 2 Some of the builds are giving following error . Please help https://travis-ci.org/cloudera/livy/jobs/238365627 - basic interactive session *** FAILED *** (5 minutes, 3 seconds) The code passed to eventually never returned normally. Attempted 305 times over 5.012684563933334 minutes. Last failure message: assertion failed: Session 0 state dead doesn't equal one of Set(idle). (LivyRestClient.scala:100) - Final session state: SessionSnapshot(0,Some(application_1496328962772_0001),dead,AppInfo(None,Some( http://testing-gce-05fca65b-482d-4c3e-8cf6-39a9f86cb83c:36897/cluster/app/application_1496328962772_0001)),Vector( at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302), at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82), at java.util.concurrent.FutureTask.run(FutureTask.java:262), at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145), at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615), at java.lang.Thread.run(Thread.java:745), , , Container exited with a non-zero exit code 15, Failing this attempt. Failing the application.)) - YARN diagnostics: Application application_1496328962772_0001 failed 1 times due to AM Container for appattempt_1496328962772_0001_000001 exited with exitCode: 15 For more detailed output, check application tracking page: http://testing-gce-05fca65b-482d-4c3e-8cf6-39a9f86cb83c:36897/cluster/app/application_1496328962772_0001Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1496328962772_0001_01_000001 Exit code: 15 Stack trace: ExitCodeException exitCode=15: at org.apache.hadoop.util.Shell.runCommand(Shell.java:582) at org.apache.hadoop.util.Shell.run(Shell.java:479) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 15 Failing this attempt. Failing the application.

…

On Thu, Jun 1, 2017 at 6:05 PM, Saisai Shao ***@***.***> wrote: Looks like Travis is broken, let me try to fix it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#334 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APZjN0tyN_mXfCBlP4VwOqltajzNxXDhks5r_rAZgaJpZM4NfH-m> .

zjffdu · 2017-06-03T02:19:04Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

+      LOG.error("Port Range format is not correct " + this.portRange);
+      throw e;
+    }
+    catch(NumberFormatException e) {


code style issue

zjffdu · 2017-06-03T02:19:26Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

-      address = config.findLocalAddress();
-    }
-    this.address = address;
+  return channel;


pralabhkumar · 2017-06-05T09:34:07Z

@zjffdu
Thanks for the review . Have done the changes as suggested .

Please review

zjffdu · 2017-06-05T09:42:06Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

+   * Get Port Numbers
+   */
+  public int[] getPortNumberAndRange() throws ArrayIndexOutOfBoundsException,
+  NumberFormatException {


indentation

zjffdu · 2017-06-05T09:42:22Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

@@ -111,6 +111,61 @@ public void run() {
  }

  /**
+   * Get Port Numbers
+   */
+  public int[] getPortNumberAndRange() throws ArrayIndexOutOfBoundsException,


change to private

zjffdu · 2017-06-05T09:42:47Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

+  /**
+   * @throws InterruptedException
+   **/
+  public Channel getChannel(int portNumber) throws BindException, InterruptedException {


zjffdu · 2017-06-05T09:43:14Z

rsc/src/main/java/com/cloudera/livy/rsc/rpc/RpcServer.java

+   **/
+  public Channel getChannel(int portNumber) throws BindException, InterruptedException {
+    Channel channel = new ServerBootstrap()
+    .group(group)


indentation

zjffdu · 2017-06-05T09:43:49Z

@jerryshao Do you have any finding why the code style issue is not detected by CI ?

jerryshao · 2017-06-05T12:22:33Z

No, I haven't got a chance to look at this issue. Let's check the style issue here manually.

pralabhkumar · 2017-06-05T13:14:06Z

@zjffdu Thanks for review . Have done all the suggested changes . Please review again :)

pralabhkumar · 2017-06-06T05:28:11Z

@zjffdu Please let me know , if we are good with the changes or further changes are needed

zjffdu · 2017-06-07T01:58:27Z

@pralabhkumar Thanks for this. Almost LGTM, one minor thing left, could you add unit test in TestRpc to verify this configuration ?

zjffdu · 2017-06-07T08:55:02Z

rsc/src/test/java/com/cloudera/livy/rsc/rpc/TestRpc.java

+    int endPort = Integer.parseInt(portRange[1]);
+    for (int index = startPort; index <= endPort; index++) {
+      RpcServer server = autoClose(new RpcServer(emptyConfig));
+      assertTrue(startPort <= server.getPort() && server.getPort() <= endPort);


Seems you are using the default value of LAUNCHER_PORT_RANGE, don't understand why using a for loop here ?

Basically I think we need to add 2 test cases. One is specifying a wrong value of LAUNCHER_PORT_RANGE and verify the correct Exception is caught, another is specifying a correct value of LAUNCHER_PORT_RANGE, and the port of server is in this range.

@zjffdu Thanks for your suggestion . Working on it , will push in an hour. :)

pralabhkumar · 2017-06-07T10:50:08Z

@zjffdu

Have created the unit test case . Please review and let me know , if its ok .

zjffdu · 2017-06-08T01:39:10Z

conf/livy-client.conf.template

@@ -55,7 +55,7 @@

 # Address for the RSC driver to connect back with it's connection info.
 # livy.rsc.launcher.address =
-# livy.rsc.launcher.port = -1
+# livy.rsc.launcher.port.range = 10000~10110


Could you add one comment for this configuration, especially about whether start/end port is inclusive and exclusive

@zjffdu
done

zjffdu · 2017-06-08T01:42:53Z

rsc/src/test/java/com/cloudera/livy/rsc/rpc/TestRpc.java

+    } catch (Exception ee) {
+      assertTrue(ee instanceof ArrayIndexOutOfBoundsException);
+    }
+    portRange = "11000~11001";


Could you use 11000~~11110 ? Because 11000~~11001 is likely to fail the unit test if these 2 port are happenly being used in the travis server.

…r unit test case

zjffdu · 2017-06-08T02:09:28Z

Thanks @pralabhkumar LGTM, wait for CI pass

pralabhkumar · 2017-06-08T02:46:07Z

@zjffdu
Thanks for help .
Build is passed . Can we please merge it .

pralabhkumar · 2017-06-08T04:04:40Z

@zjffdu Please let me know if anything else is needed from my side. :)

pralabhkumar · 2017-06-08T08:07:16Z

@zffdu . Thanks for merging it and really thankful for providing help through out the process.

Can u please close the jira LIVY-337

https://issues.cloudera.org/browse/LIVY-337

Thanks for your support.

Code changes in RPCserver for user provided port

bc8f2ac

pralabhkumar force-pushed the develop branch from a3e28da to bc8f2ac Compare May 20, 2017 16:40

pralabhkumar added 5 commits May 20, 2017 18:09

Indentation Changes

968f584

Indentation Changes

8e3c0ef

Indentation Changes

a8c53b8

Indentation Changes

94d23e5

Configuring Port Range

994ac16

Documentation Changed

238d238

pralabhkumar added 3 commits May 29, 2017 06:54

launcher.port.range will take care of launching RPC

951739c

Checkstyle changes

dbaf50a

Checkstyle changes

4a7e219

zjffdu reviewed May 31, 2017

View reviewed changes

zjffdu reviewed Jun 3, 2017

View reviewed changes

pralabhkumar added 2 commits June 5, 2017 08:33

Indentation Changes & port range in livy-client.conf.template

edbf6c9

Indentation changes

2c4189c

zjffdu reviewed Jun 5, 2017

View reviewed changes

pralabhkumar added 2 commits June 5, 2017 10:24

Changed visibilty of method private

2b91398

Indentation Changes

a8e29d1

Indenetation Changes

e59e9a7

pralabhkumar added 2 commits June 7, 2017 08:42

Unit test case to test port range

59683d3

Checkstyle changes

57b6c51

zjffdu reviewed Jun 7, 2017

View reviewed changes

Unit test case for port range

a22222e

zjffdu reviewed Jun 8, 2017

View reviewed changes

Added comment for Port Range Configuration and increase port range fo…

3f58233

…r unit test case

zjffdu merged commit 9ae24d0 into cloudera:master Jun 8, 2017

Livy:337 Binding RPCServer to user provided port and not random port #334

Livy:337 Binding RPCServer to user provided port and not random port #334

Conversation

pralabhkumar commented May 18, 2017

codecov-io commented May 21, 2017 • edited Loading

Codecov Report

pralabhkumar commented May 21, 2017 • edited Loading

pralabhkumar commented May 22, 2017

pralabhkumar commented May 24, 2017

pralabhkumar commented May 26, 2017

zjffdu commented May 26, 2017 • edited Loading

pralabhkumar commented May 28, 2017

pralabhkumar commented May 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zjffdu commented May 31, 2017 • edited Loading

jerryshao commented May 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pralabhkumar commented May 31, 2017

pralabhkumar commented Jun 1, 2017 • edited Loading

pralabhkumar commented Jun 1, 2017

jerryshao commented Jun 1, 2017

pralabhkumar commented Jun 1, 2017

pralabhkumar commented Jun 1, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pralabhkumar commented Jun 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zjffdu commented Jun 5, 2017

jerryshao commented Jun 5, 2017

pralabhkumar commented Jun 5, 2017

pralabhkumar commented Jun 6, 2017

zjffdu commented Jun 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pralabhkumar commented Jun 7, 2017 • edited Loading

zjffdu Jun 8, 2017 • edited Loading

Choose a reason for hiding this comment

pralabhkumar Jun 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zjffdu commented Jun 8, 2017

pralabhkumar commented Jun 8, 2017 • edited Loading

pralabhkumar commented Jun 8, 2017

pralabhkumar commented Jun 8, 2017

codecov-io commented May 21, 2017 •

edited

Loading

pralabhkumar commented May 21, 2017 •

edited

Loading

zjffdu commented May 26, 2017 •

edited

Loading

zjffdu commented May 31, 2017 •

edited

Loading

pralabhkumar commented Jun 1, 2017 •

edited

Loading

pralabhkumar commented Jun 7, 2017 •

edited

Loading

zjffdu Jun 8, 2017 •

edited

Loading

pralabhkumar Jun 8, 2017 •

edited

Loading

pralabhkumar commented Jun 8, 2017 •

edited

Loading