Add timing projection #219

betanalpha · 2013-09-17T21:22:47Z

Help alleviate overambitious users.

stan-buildbot · 2013-09-17T21:22:58Z

code review

Can one of the admins verify this patch?

bob-carpenter · 2013-09-18T00:16:45Z

This is to let a user know how long it's going to take? Presumably if each iteration takes a long time, this won't just scroll off the screen --- isn't there some kind of pacifier output per iteration anyway?

I love the idea of having expected completion times if we can estimate them even remotely reliably.

betanalpha · 2013-09-18T07:37:41Z

This only prints at the beginning of a run -- the idea is to make a user aware that a model might take days to run, instead of the seconds/minutes they might be expecting, and hopefully encourage them to restart with a simpler model.

On Sep 18, 2013, at 1:16 AM, Bob Carpenter [email protected] wrote:

This is to let a user know how long it's going to take? Presumably if each iteration takes a long time, this won't just scroll off the screen --- isn't there some kind of pacifier output per iteration anyway?

I love the idea of having expected completion times if we can estimate them even remotely reliably.

—
Reply to this email directly or view it on GitHub.

bob-carpenter · 2013-09-18T08:04:36Z

I'm OK with this. It's a simple change.

Bob

On 9/18/13 3:37 AM, Michael Betancourt wrote:

This only prints at the beginning of a run -- the idea is to make a user aware that a model might take days to run,
instead of the seconds/minutes they might be expecting, and hopefully encourage them to restart with a simpler model.

On Sep 18, 2013, at 1:16 AM, Bob Carpenter [email protected] wrote:

This is to let a user know how long it's going to take? Presumably if each iteration takes a long time, this won't
just scroll off the screen --- isn't there some kind of pacifier output per iteration anyway?

I love the idea of having expected completion times if we can estimate them even remotely reliably.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #219 (comment).

syclik · 2013-09-19T04:10:27Z

Jenkins, ok to test.

stan-buildbot · 2013-09-19T19:27:38Z

Test FAILed.

Refer to this link for build results: http://d1m1s1b1.stat.columbia.edu:8080/job/Stan%20Github%20Pull%20Requests/213/

bob-carpenter · 2013-09-19T19:31:13Z

This looks like it's breaking because offsets into output are being hard coded rather than matched by name. Is there a way to make the tests match by content rather than line number? Matching by line number is going to be super brittle for any little changes we do in the output like this pull request.

betanalpha · 2013-09-24T12:00:17Z

Tests fixed.

stan-buildbot · 2013-09-25T05:18:40Z

Test PASSed.
Refer to this link for build results: http://d1m1s1b1.stat.columbia.edu:8080/job/Stan%20Github%20Pull%20Requests/216/

syclik · 2013-11-01T22:34:40Z

Jenkins, retest this please.

syclik · 2013-11-01T22:35:01Z

This should have been merged a while ago, just want to make sure it plays nicely with develop.

syclik · 2013-11-11T22:23:32Z

Jenkins, retest this please.

syclik · 2013-11-15T19:54:09Z

Jenkins, retest this please.

syclik · 2013-11-26T19:38:51Z

Jenkins, retest this please.

stan-buildbot · 2013-11-27T06:03:06Z

Test PASSed.
Refer to this link for build results: http://d1m1s1b1.stat.columbia.edu:8080/job/Stan%20Github%20Pull%20Requests/292/

syclik · 2013-11-27T06:15:46Z

The output looks like this:

> ./models/basic_distributions/normal sample
Gradient evaluation took 7e-06 seconds
1000 transitions using 10 leapfrog steps per transition would take 0.07 seconds.
Suit your expectations accordingly!

 method = sample (Default)
   sample
     num_samples = 1000 (Default)
     num_warmup = 1000 (Default)
     save_warmup = 0 (Default)
     thin = 1 (Default)
     adapt
       engaged = 1 (Default)
       gamma = 0.050000000000000003 (Default)
       delta = 0.65000000000000002 (Default)
       kappa = 0.75 (Default)
       t0 = 10 (Default)
     algorithm = hmc (Default)
       hmc
         engine = nuts (Default)
           nuts
             max_depth = 10 (Default)
         metric = diag_e (Default)
         stepsize = 1 (Default)
         stepsize_jitter = 0 (Default)
 id = 0 (Default)
 data =  (Default)
 init = 2 (Default)
 random
   seed = 2673486659
 output
   file = samples.csv (Default)
   append_sample = 0 (Default)
   diagnostic_file =  (Default)
   append_diagnostic = 0 (Default)
   refresh = 100 (Default)

Iteration:    1 / 2000 [  0%]  (Warmup)
Iteration:  100 / 2000 [  5%]  (Warmup)
Iteration:  200 / 2000 [ 10%]  (Warmup)
Iteration:  300 / 2000 [ 15%]  (Warmup)
...

I think I would prefer the timing to be after the output block indicating how the command line was called. The output gets buried deep. I could be convinced otherwise.

betanalpha · 2013-11-27T14:25:49Z

My concern with moving it would be making it less obvious to
the user -- I want it front an center.

On Nov 27, 2013, at 6:15 AM, Daniel Lee [email protected] wrote:

The output looks like this:

./models/basic_distributions/normal sample
Gradient evaluation took 7e-06 seconds
1000 transitions using 10 leapfrog steps per transition would take 0.07 seconds.
Suit your expectations accordingly!

method = sample (Default)
sample
num_samples = 1000 (Default)
num_warmup = 1000 (Default)
save_warmup = 0 (Default)
thin = 1 (Default)
adapt
engaged = 1 (Default)
gamma = 0.050000000000000003 (Default)
delta = 0.65000000000000002 (Default)
kappa = 0.75 (Default)
t0 = 10 (Default)
algorithm = hmc (Default)
hmc
engine = nuts (Default)
nuts
max_depth = 10 (Default)
metric = diag_e (Default)
stepsize = 1 (Default)
stepsize_jitter = 0 (Default)
id = 0 (Default)
data = (Default)
init = 2 (Default)
random
seed = 2673486659
output
file = samples.csv (Default)
append_sample = 0 (Default)
diagnostic_file = (Default)
append_diagnostic = 0 (Default)
refresh = 100 (Default)

Iteration: 1 / 2000 0%
Iteration: 100 / 2000 5%
Iteration: 200 / 2000 10%
Iteration: 300 / 2000 15%
...

I think I would prefer the timing to be after the output block indicating how the command line was called. The output gets buried deep. I could be convinced otherwise.

—
Reply to this email directly or view it on GitHub.

bob-carpenter · 2013-11-27T16:07:32Z

I like the expected timing.

It's probably going to be inaccurate for probability
functions with iterative algorithms like integrators.

My preferences about ordering match Michael's -- I like
the timing right up front (of course then the danger is
that it scrolls off the screen given the verbosity of the
command-line output [which I really like!]).

I think it'd help to give it in terms of number
of leapfrog steps and tree depth, because the latter is
what we report. So maybe use 7 leapfrog steps, which corresponds
to a tree depth of 2 once we get the conforming definition
(but probably 3 now if that hasn't been fixed yet).

Minor point: "suit your expectations" is synonymous with
"fulfill your expectations" or "match your expectations",
so it's awkard as a command; I think either "set your expections"
or "adjust your expectations" would be better.

Bob

On 11/27/13, 9:25 AM, Michael Betancourt wrote:

My concern with moving it would be making it less obvious to
the user -- I want it front an center.

On Nov 27, 2013, at 6:15 AM, Daniel Lee [email protected] wrote:

The output looks like this:

./models/basic_distributions/normal sample
Gradient evaluation took 7e-06 seconds
1000 transitions using 10 leapfrog steps per transition would take 0.07 seconds.
Suit your expectations accordingly!

method = sample (Default)
sample
num_samples = 1000 (Default)
num_warmup = 1000 (Default)
save_warmup = 0 (Default)
thin = 1 (Default)
adapt
engaged = 1 (Default)
gamma = 0.050000000000000003 (Default)
delta = 0.65000000000000002 (Default)
kappa = 0.75 (Default)
t0 = 10 (Default)
algorithm = hmc (Default)
hmc
engine = nuts (Default)
nuts
max_depth = 10 (Default)
metric = diag_e (Default)
stepsize = 1 (Default)
stepsize_jitter = 0 (Default)
id = 0 (Default)
data = (Default)
init = 2 (Default)
random
seed = 2673486659
output
file = samples.csv (Default)
append_sample = 0 (Default)
diagnostic_file = (Default)
append_diagnostic = 0 (Default)
refresh = 100 (Default)

Iteration: 1 / 2000 0%
Iteration: 100 / 2000 5%
Iteration: 200 / 2000 10%
Iteration: 300 / 2000 15%
...

I think I would prefer the timing to be after the output block indicating how the command line was called. The output
gets buried deep. I could be convinced otherwise.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub #219 (comment).

stan-buildbot · 2013-12-07T01:28:35Z

src/stan/gm/command.hpp

+      std::cout << "Gradient evaluation took " << deltaT << " seconds" << std::endl;
+      std::cout << "1000 transitions using 10 leapfrog steps per transition would take "
+                << 1e4 * deltaT << " seconds." << std::endl;
+      std::cout << "Suit your expectations accordingly!" << std::endl << std::endl;


@betanalpha I'll let you choose what the final text is, but pick between:
"set your expections" or "adjust your expectations"

Or let me know which one you want and I'll take care of it. As far as I'm concerned, it's good to go once this gets changed. (I'd still prefer the info to come after the argument specification, but I'm good with it as-is.)

Sorry -- that was from me. I was logged in as stan-buildbot.

On Fri, Dec 6, 2013 at 8:28 PM, Stan buildbot [email protected]:

In src/stan/gm/command.hpp:

@@ -499,6 +499,22 @@

}

// Check timing

clock_t start = clock();

std::vector<double> init_grad;

stan::model::log_prob_grad<true, true>(model,

cont_params, disc_params, init_grad,

&std::cout);

clock_t end = clock();

double deltaT = (double)(end - start) / CLOCKS_PER_SEC;

std::cout << "Gradient evaluation took " << deltaT << " seconds" << std::endl;

std::cout << "1000 transitions using 10 leapfrog steps per transition would take "

<< 1e4 \* deltaT << " seconds." << std::endl;

std::cout << "Suit your expectations accordingly!" << std::endl << std::endl;

@betanalpha https://github.com/betanalpha I'll let you choose what the
final text is, but pick between:

"set your expections" or "adjust your expectations"

Or let me know which one you want and I'll take care of it. As far as I'm
concerned, it's good to go once this gets changed. (I'd still prefer the
info to come after the argument specification, but I'm good with it as-is.)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/219/files#r8179476
.

Updated to "adjust your expectations".

On Dec 6, 2013, at 8:28 PM, Stan buildbot [email protected] wrote:

In src/stan/gm/command.hpp:

@@ -499,6 +499,22 @@

}

// Check timing

clock_t start = clock();

std::vector<double> init_grad;

stan::model::log_prob_grad<true, true>(model,

cont_params, disc_params, init_grad,

&std::cout);

clock_t end = clock();

double deltaT = (double)(end - start) / CLOCKS_PER_SEC;

std::cout << "Gradient evaluation took " << deltaT << " seconds" << std::endl;

std::cout << "1000 transitions using 10 leapfrog steps per transition would take "

<< 1e4 \* deltaT << " seconds." << std::endl;

std::cout << "Suit your expectations accordingly!" << std::endl << std::endl;

@betanalpha I'll let you choose what the final text is, but pick between:
"set your expections" or "adjust your expectations"

Or let me know which one you want and I'll take care of it. As far as I'm concerned, it's good to go once this gets changed. (I'd still prefer the info to come after the argument specification, but I'm good with it as-is.)

—
Reply to this email directly or view it on GitHub.

syclik · 2013-12-07T15:04:21Z

Jenkins, retest this please.

syclik · 2013-12-07T15:07:00Z

@betanalpha, I just realized the manual should change reflecting these changes, right?

betanalpha · 2013-12-07T15:27:45Z

Gah, yeah. Although it should only be in a few places.

On Dec 7, 2013, at 10:07 AM, Daniel Lee [email protected] wrote:

@betanalpha, I just realized the manual should change reflecting these changes, right?

—
Reply to this email directly or view it on GitHub.

syclik · 2013-12-07T15:29:25Z

I can take care of it. I don't think we need to do a full test run after
the doc changes.

On Sat, Dec 7, 2013 at 10:27 AM, Michael Betancourt <
[email protected]> wrote:

Gah, yeah. Although it should only be in a few places.

On Dec 7, 2013, at 10:07 AM, Daniel Lee [email protected]
wrote:

@betanalpha, I just realized the manual should change reflecting these
changes, right?

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/219#issuecomment-30057151
.

syclik · 2013-12-07T20:26:40Z

@betanalpha, I take it back. I'll let you write in how you want it written up.

Having the timing come in before the echoed parameters changes a lot more of the manual than after. I think I'd prefer to have you write it up how you're comfortable.

In introduction.tex, in "Sampling from the Model," we've got this text:

Whether the command is run in Windows, Linux, or on the Mac, the
output is the same. First, the parameters are echoed to the standard output,
which shows up on the terminal as follows.
with the echoed command-line parameters following directly after.

In commands.tex, in "Command-Line Option Examples," the multiple figures needs to change. Most of the manual is written with the same assumption I've made: on any valid Stan run (whether it's optimize or sample), it will echo the parameters. I'm ok with it coming up front, we just need to fix the manual.

bob-carpenter · 2013-12-07T21:20:55Z

If you want to just dump the pieces of the manual that
need to change in here:

#325

I'll do them when I take a final pass at the manual,
which I plan to do after flexible adaptation gets
merged.

Bob

On 12/7/13, 3:26 PM, Daniel Lee wrote:

@betanalpha https://github.com/betanalpha, I take it back. I'll let you write in how you want it written up.

Having the timing come in /before/ the echoed parameters changes a lot more of the manual than after. I think I'd prefer
to have you write it up how you're comfortable.

In introduction.tex, in "Sampling from the Model," we've got this text:
Whether the command is run in Windows, Linux, or on the Mac, the
output is the same. First, the parameters are echoed to the standard output,
which shows up on the terminal as follows.
with the echoed command-line parameters following directly after.
In commands.tex, in "Command-Line Option Examples," the multiple figures needs to change. Most of the manual is written
with the same assumption I've made: on /any/ valid Stan run (whether it's optimize or sample), it will echo the
parameters. I'm ok with it coming up front, we just need to fix the manual.

—
Reply to this email directly or view it on GitHub #219 (comment).

syclik · 2013-12-07T21:22:35Z

Ok. I'll merge now and we'll add manual changes to that fix.
On Dec 7, 2013 4:20 PM, "Bob Carpenter" [email protected] wrote:

If you want to just dump the pieces of the manual that
need to change in here:

#325

I'll do them when I take a final pass at the manual,
which I plan to do after flexible adaptation gets
merged.

Bob

On 12/7/13, 3:26 PM, Daniel Lee wrote:

@betanalpha https://github.com/betanalpha, I take it back. I'll let
you write in how you want it written up.

Having the timing come in /before/ the echoed parameters changes a lot
more of the manual than after. I think I'd prefer
to have you write it up how you're comfortable.

In introduction.tex, in "Sampling from the Model," we've got this text:

Whether the command is run in Windows, Linux, or on the Mac, the
output is the same. First, the parameters are echoed to the standard
output,
which shows up on the terminal as follows.
with the echoed command-line parameters following directly after.

In commands.tex, in "Command-Line Option Examples," the multiple figures
needs to change. Most of the manual is written
with the same assumption I've made: on /any/ valid Stan run (whether
it's optimize or sample), it will echo the
parameters. I'm ok with it coming up front, we just need to fix the
manual.

—
Reply to this email directly or view it on GitHub <
https://github.com/stan-dev/stan/pull/219#issuecomment-30063938>.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/219#issuecomment-30065148
.

stan-buildbot · 2013-12-08T02:09:43Z

Test PASSed.
Refer to this link for build results: http://d1m1s1b1.stat.columbia.edu:8080/job/Stan%20Github%20Pull%20Requests/313/

Add timing projection

Add timing projection

0e53c3b

Fix argument config test

e955fdd

stan-buildbot reviewed Dec 7, 2013
View reviewed changes

Tweak timing message

4d5157b

ghost assigned syclik Dec 7, 2013

Merge branch 'develop' into feature/timing_warning

02d2008

syclik added a commit that referenced this pull request Dec 8, 2013

Merge pull request #219 from stan-dev/feature/timing_warning

0767e52

Add timing projection

syclik merged commit 0767e52 into develop Dec 8, 2013

syclik deleted the feature/timing_warning branch December 8, 2013 04:15

syclik mentioned this pull request Dec 8, 2013

manual changes for next release #325

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add timing projection #219

Add timing projection #219

betanalpha commented Sep 17, 2013

stan-buildbot commented Sep 17, 2013

bob-carpenter commented Sep 18, 2013

betanalpha commented Sep 18, 2013

bob-carpenter commented Sep 18, 2013

syclik commented Sep 19, 2013

stan-buildbot commented Sep 19, 2013

bob-carpenter commented Sep 19, 2013

betanalpha commented Sep 24, 2013

stan-buildbot commented Sep 25, 2013

syclik commented Nov 1, 2013

syclik commented Nov 1, 2013

syclik commented Nov 11, 2013

syclik commented Nov 15, 2013

syclik commented Nov 26, 2013

stan-buildbot commented Nov 27, 2013

syclik commented Nov 27, 2013

betanalpha commented Nov 27, 2013

bob-carpenter commented Nov 27, 2013

stan-buildbot Dec 7, 2013

syclik Dec 7, 2013

betanalpha Dec 7, 2013

syclik commented Dec 7, 2013

syclik commented Dec 7, 2013

betanalpha commented Dec 7, 2013

syclik commented Dec 7, 2013

syclik commented Dec 7, 2013

bob-carpenter commented Dec 7, 2013

syclik commented Dec 7, 2013

stan-buildbot commented Dec 8, 2013

Add timing projection #219

Add timing projection #219

Conversation

betanalpha commented Sep 17, 2013

stan-buildbot commented Sep 17, 2013

bob-carpenter commented Sep 18, 2013

betanalpha commented Sep 18, 2013

bob-carpenter commented Sep 18, 2013

syclik commented Sep 19, 2013

stan-buildbot commented Sep 19, 2013

bob-carpenter commented Sep 19, 2013

betanalpha commented Sep 24, 2013

stan-buildbot commented Sep 25, 2013

syclik commented Nov 1, 2013

syclik commented Nov 1, 2013

syclik commented Nov 11, 2013

syclik commented Nov 15, 2013

syclik commented Nov 26, 2013

stan-buildbot commented Nov 27, 2013

syclik commented Nov 27, 2013

betanalpha commented Nov 27, 2013

bob-carpenter commented Nov 27, 2013

stan-buildbot Dec 7, 2013

Choose a reason for hiding this comment

syclik Dec 7, 2013

Choose a reason for hiding this comment

betanalpha Dec 7, 2013

Choose a reason for hiding this comment

syclik commented Dec 7, 2013

syclik commented Dec 7, 2013

betanalpha commented Dec 7, 2013

syclik commented Dec 7, 2013

syclik commented Dec 7, 2013

bob-carpenter commented Dec 7, 2013

syclik commented Dec 7, 2013

stan-buildbot commented Dec 8, 2013