# Initial feedback
**URL:** https://discourse.pumas.ai/t/initial-feedback/31
**Category:** Uncategorized
**Created:** [August 1, 2019, 6:13pm UTC](https://discourse.pumas.ai/t/initial-feedback/31 "2019-08-01T18:13:09Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@ronkeizer](https://discourse.pumas.ai/u/ronkeizer)
#### Post date: [August 1, 2019, 6:13pm UTC](https://discourse.pumas.ai/t/initial-feedback/31/1 "2019-08-01T18:13:09Z")
Overall good first experience, basic functionality works, looks very promising. Below some initial feedback / questions.
## Fitting
- Overall performance is fast, fits feels broadly similar in speed to NONMEM (tested only on very simple problems so far). Will do more benchmarking later.
- Loading the Pumas package does take some time, about 35secs on a 2017 i7 MBP, but not a problem in most use-cases since you only have to recompile the package once in each session.
- Is there any way I can get information on the iterations / warnings / etc that is going on when doing a fit? In the JuliaPro interface I’m only seeing the “cogs” icon when something is running.
- Although time-varying covariates seem to be parsed fine using `read-pumas()`, when using this parse dataset in the `fit()` it complains with `MethodError: no method matching ^(::Array{Float64,1}, ::Float64)` which seems to indicate vectors of covariates are not allowed yet? Am I doing something wrong, or is this a feature that is coming later?
## VPC
I realize this functionality is alpha, but here are some comments anyhow:
- why are there two light-red areas in the VPC plot? I guess one is for the prediction interval for the median (of the simulated data) and the other the uncertainty of the observed data? If so, I wouldn’t do that, at least not as default, as it makes the plot too cluttered and hard to interpret.
- I suppose the red line is the median of the observed, but what are the pink dots then? Most pharmacometricians will intuitively think these are the actual observations, but that doesn’t seem to be the case?
- binning is an essential feature of the VPC, I don’t see that as an option yet? Should be easy to add at least simple k-means clustering and custom binning options.
- personal preference: i would keep the colors of the prediction intervals (e.g. 5th and 95 percentile) the same (now blue and green), as they provide similar types of information.
## Misc suggestions:
- Would you be willing to share the development roadmap, or at least a broad version of it? This would help us decide when / whether this is likely to become a possible alternative in production use for us.
- in general, as someone who is not too very familiar with Julia yet, it requires quite a bit of tinkering to get started beyond the few provided examples (e.g. using NONMEM datasets). In the R world, it is common that R package developers guide the e.g. by adding useful error / warning message that catch common mistakes when calling functions. Not sure if this is common in the Julia world, but would make transition easier for novices.
- what exactly is the difference is between `read_pumas()` and `process_nmtran()`? After some tinkering I managed to read a NONMEM file with `read_pumas`, it seems the latter is not available yet? I think it would be useful for Julia novices to show with a simple example how to get a NONMEM dataset (csv) into Julia and fit it.
- BayesMCMC: can you add some more detailed info in the manual on how to run using BayesMCMC, just swapping `Pumas.FOCEI` for `BayesMCMC` doesn’t work. Questions e.g. :
- how do I specify the priors?
- what MCMC samplers are available, which one is used by default?
- does this use a separate library in the backend (`MCMC`?), does it need to be installed?
---
### Author:  [@vijay](https://discourse.pumas.ai/u/vijay)
#### Post date: [August 1, 2019, 7:15pm UTC](https://discourse.pumas.ai/t/initial-feedback/31/2 "2019-08-01T19:15:36Z")
Dear Ron,
Thank you for the detailed feedback. While I will take a shot at responding to most of the things, I am sure our team will give more details.
## Fitting
> - Overall performance is fast, fits feels broadly similar in speed to NONMEM (tested only on very simple problems so far). Will do more benchmarking later.
We have an entire suite of benchmarks that we will be putting out soon. We are tinkering on some internals and once that is done, we will be able run the comparisons easily.
> - Loading the Pumas package does take some time, about 35secs on a 2017 i7 MBP, but not a problem in most use-cases since you only have to recompile the package once in each session.
Yes, we realize this compile-time issue. @pkofod and @ChrisRackauckas are currently working on cutting them down. It is WIP
> - Is there any way I can get information on the iterations / warnings / etc that is going on when doing a fit? In the JuliaPro interface I’m only seeing the “cogs” icon when something is running.
In our initial releases we turned off the output printing by default. @pkofod is working on getting that back after our discussions at JuliaCon. But, if you would like to see the traces after fitting, you can access the fields of the `fit` object. E.g. if you have `myfit = fit(..)`, then `myfit.model`, `myfit.optim.trace` etc.. We have documentation coming out on that soon.
> - Although time-varying covariates seem to be parsed fine using `read-pumas()` , when using this parse dataset in the `fit()` it complains with `MethodError: no method matching ^(::Array{Float64,1}, ::Float64)` which seems to indicate vectors of covariates are not allowed yet? Am I doing something wrong, or is this a feature that is coming later?
Time-varying covariates work with ODE’s as of now. There is a special syntax in the `@model` macro to define time-varying covaraites. Again, a tutorial is ready and will put this out soon. Time-varying covariates on analytical solutions is WIP and we anticipate that to be ready by ACoP
## VPC
> I realize this functionality is alpha, but here are some comments anyhow:
There is much to be done here, and we just got started. Your experience and feedback would be really helpful here. We will answer these questions separately.
- why are there two light-red areas in the VPC plot? I guess one is for the prediction interval for the median (of the simulated data) and the other the uncertainty of the observed data? If so, I wouldn’t do that, at least not as a default, as it makes the plot too cluttered and hard to interpret.
- I suppose the red line is the median of the observed, but what are the pink dots then? Most pharmacometricians will intuitively think these are the actual observations, but that doesn’t seem to be the case?
- binning is an essential feature of the VPC, I don’t see that as an option yet? Should be easy to add at least simple k-means clustering and custom binning options.
- personal preference: i would keep the colors of the prediction intervals (e.g. 5th and 95 percentile) the same (now blue and green), as they provide similar types of information.
## Misc suggestions:
> - Would you be willing to share the development roadmap, or at least a broad version of it? This would help us decide when / whether this is likely to become a possible alternative in production use for us.
Yes, we plan to do that before ACoP.
> - in general, as someone who is not too very familiar with Julia yet, it requires quite a bit of tinkering to get started beyond the few provided examples (e.g. using NONMEM datasets). In the R world, it is common that R package developers guide the e.g. by adding useful error / warning message that catch common mistakes when calling functions. Not sure if this is common in the Julia world, but would make transition easier for novices.
This has been on our to-do list. Providing appropriate error messages is hard and comes with user feedback and testing. We are working on getting more and more simple errors out, but this may take a little bit of time. Maybe @ChrisRackauckas or @andreasnoack can address this question better.
> - what exactly is the difference is between `read_pumas()` and `process_nmtran()` ? After some tinkering I managed to read a NONMEM file with `read_pumas` , it seems the latter is not available yet? I think it would be useful for Julia novices to show with a simple example how to get a NONMEM dataset (csv) into Julia and fit it.
While we started with `process_nmtran` which does everything that nmtran datasets require, we realized that a lot of the requirements can be overriden by easier Julia based features. `read_pumas` will be compatible with nmtran style datasets, but will eventually morph into its own style of data set preparations that will utilize the power of Julia. @ChrisRackauckas can expand more here.
> - BayesMCMC: can you add some more detailed info in the manual on how to run using BayesMCMC, just swapping `Pumas.FOCEI` for `BayesMCMC` doesn’t work. Questions e.g. :
> - how do I specify the priors?
> - what MCMC samplers are available, which one is used by default?
> - does this use a separate library in the backend ( `MCMC` ?), does it need to be installed?
Documentation on this is sparse and there is some work going on currently. To be honest, the only change here is that in the `@param` block, you direclty specify the distribution of the prior instead of the bounds. And then call the `Pumas.BayesMCMC()` in the `fit` function. We are working on post-processing the Bayes fits and other details. We plan to have a ready to use Bayes in month or two.
Please look out for the roadmap for other details.
---
### Author:  [@ChrisRackauckas](https://discourse.pumas.ai/u/ChrisRackauckas)
#### Post date: [August 2, 2019, 11:29am UTC](https://discourse.pumas.ai/t/initial-feedback/31/3 "2019-08-02T11:29:59Z")
> [@vijay](#):
>
> We have an entire suite of benchmarks that we will be putting out soon. We are tinkering on some internals and once that is done, we will be able run the comparisons easily.
Indeed, the benchmarks will be automatically runnable and updated to each version. I hope we can get those out by ACoP.
> [@vijay](#):
>
> Yes, we realize this compile-time issue. @pkofod and @ChrisRackauckas are currently working on cutting them down. It is WIP
The JuliaPro registry currently has old versions of the DifferentialEquations packages which introduced some compile time regressions due to interactions with Cassette. These were subsequently made optional, dropping the DelayDiffEq (a dependency of Pumas) compile time from 55 seconds to 0.2 on Win32 platforms. Source: [Slow compile times · Issue #125 · SciML/DelayDiffEq.jl · GitHub](https://github.com/JuliaDiffEq/DelayDiffEq.jl/issues/125) . Additionally, some inference fixes dropped the total test time of DelayDiffEq.jl from 44 minutes on Julia 1.0 to 13 minutes on Julia 1.3, with a good portion of that being compile time (since the delay differential equation solvers are the heaviest on compile time), and we will probably drop the direct dependency on them in the next Pumas release as well (and instead utilize abstract type handling).
There is another compile time issue in Pumas which I think I am tracking down to AdvancedHMC.jl, and we’ll be working that out similarly.
> [@vijay](#):
>
> Time-varying covariates work with ODE’s as of now. There is a special syntax in the `@model` macro to define time-varying covaraites. Again, a tutorial is ready and will put this out soon. Time-varying covariates on analytical solutions is WIP and we anticipate that to be ready by ACoP
I anticipate finishing the PR by the end of next week: [https://github.com/PumasAI/Pumas.jl/pull/607](https://github.com/PumasAI/Pumas.jl/pull/607) . Of course, it’ll take longer to release.
> [@vijay](#):
>
> This has been on our to-do list. Providing appropriate error messages is hard and comes with user feedback and testing. We are working on getting more and more simple errors out, but this may take a little bit of time. Maybe @ChrisRackauckas or @andreasnoack can address this question better.
Not only are we adding more error messages on the Pumas level, but also looking to change the default error messages of Julia itself to be more friendly to new users. I’ve discussed my thoughts on this in the compiler triage calls, and they were mentioned in Jeff Bezanson’s JuliaCon talk, so it seems they are being taken seriously and will likely get into one of the next Julia versions.
> [@vijay](#):
>
> While we started with `process_nmtran` which does everything that nmtran datasets require, we realized that a lot of the requirements can be overriden by easier Julia based features. `read_pumas` will be compatible with nmtran style datasets, but will eventually morph into its own style of data set preparations that will utilize the power of Julia. @ChrisRackauckas can expand more here.
`process_nmtran` isn’t a function that exists in the released versions. The name change occurred since what we parse to isn’t exactly the NMTRAN format, but we do have things in our parser that allow for NMTRAN files to work with our parser. There are many computational advantages that we get by using our more column-based system.
> [@vijay](#):
>
> Documentation on this is sparse and there is some work going on currently. To be honest, the only change here is that in the `@param` block, you direclty specify the distribution of the prior instead of the bounds. And then call the `Pumas.BayesMCMC()` in the `fit` function. We are working on post-processing the Bayes fits and other details. We plan to have a ready to use Bayes in month or two.
We used DynamicHMC.jl as the backend for the release version, but this was not producing stable estimates and thus we are changing to AdvancedHMC.jl. Not only is this faster, but it requires less user input and is very robust. We haven’t really detailed the Bayes functionality yet since we knew we were going to swap this out.
> [@ronkeizer](#):
>
> - Would you be willing to share the development roadmap, or at least a broad version of it? This would help us decide when / whether this is likely to become a possible alternative in production use for us.
I will sharing a whole lot on this quite soon 😄 though it’s blocked right now for various reasons. Expect a very fun blog post.
---
### Author:  [@ronkeizer](https://discourse.pumas.ai/u/ronkeizer)
#### Post date: [August 2, 2019, 4:55pm UTC](https://discourse.pumas.ai/t/initial-feedback/31/4 "2019-08-02T16:55:55Z")
Thanks for the detailed feedback Vijay and Chris, this is very helpful. Will be following upcoming releases and blog posts closely!
---
### Author:  [@vijay](https://discourse.pumas.ai/u/vijay)
#### Post date: [August 2, 2019, 6:18pm UTC](https://discourse.pumas.ai/t/initial-feedback/31/5 "2019-08-02T18:18:42Z")
We expect at least one release every month till ACoP. Blog posts will go along with the release. Will keep you posted!