# JH Plugin Question

**URL:** <https://discourse.pumas.ai/t/jh-plugin-question/652>\
**Category:** Usage\
**Created:** [March 17, 2022, 9:57pm UTC](https://discourse.pumas.ai/t/jh-plugin-question/652 "2022-03-17T21:57:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![donaldlee3](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pumas.ai/donaldlee3/32/165_2.png) [@donaldlee3](https://discourse.pumas.ai/u/donaldlee3)\
**Post date:** [March 17, 2022, 9:57pm UTC](https://discourse.pumas.ai/t/jh-plugin-question/652/1 "2022-03-17T21:57:07Z")

</div>

@andreasnoack I wanted to use the plugin for a large job that would require TB’s of memory to run in one iteration. I am planning on looping through the process like 1000x and saving what I need each time through the loop. Is this possible?

The following is code is what I had to save on the plugin before. How would I modify it to save at the end of each time through a loop?

```auto
open("results_500_relF_T_3_1.jls", "w") do f
    serialize(f, Dict("MAPE_relF_summaries" => MAPE_relF_summaries, "MAPE_Vc_summaries" => MAPE_Vc_summaries, "MAPE_CL_summaries" => MAPE_CL_summaries,
                        "MPE_relF_summaries" => MPE_relF_summaries, "MPE_Vc_summaries" => MPE_Vc_summaries, "MPE_CL_summaries" => MPE_CL_summaries,
                        "MAPE_BE_cmax_summaries" => MAPE_BE_cmax_summaries, "MAPE_BE_auc_summaries" => MAPE_BE_auc_summaries,
                        "MPE_BE_cmax_summaries" => MPE_BE_cmax_summaries, "MPE_BE_auc_summaries" => MPE_BE_auc_summaries,
                        "BE_power_df" => BE_power_df))
end

ENV["RESULTS_FILE"] = "results_500_relF_T_3_1.jls"

```

Also, is there an efficient to manage the memory through each loop? I’m used to being able to easily clear the workspace in MATLAB. Should I do `variable = nothing` for all the large memory objects at the end of each loop?

And, should this outer loop be distributed with `@everywhere` as well?

Please let me know if there might be any other important considerations with this approach that I haven’t mentioned.

Thank you.

---

<div class="post-metadata">

**Author:** ![andreasnoack](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pumas.ai/andreasnoack/32/22_2.png) [@andreasnoack](https://discourse.pumas.ai/u/andreasnoack)\
**Post date:** [March 18, 2022, 2:51pm UTC](https://discourse.pumas.ai/t/jh-plugin-question/652/2 "2022-03-18T14:51:52Z")

</div>

You should be able to organize your script as a loop where you save a result file in each iteration of the loop to a temporary location on the server and then, at the end, you tar up the results and mark the file for upload. I.e. something like this (warning: untested)

```julia
tarname = join(tempname(), ".tar.gz")
resultsdir = joinpath(tempdir(), "results")
mkpath(resultsdir)

ENV["RESULTS_FILE"] = tarname

for i in 1:1000
    result = ...do stuff...

    open(join(tempname(parent=resultsdir), ".jls"), "w") do f
        serialize(f, ...)
    end
end

run(`tar -czvf $tarname $resultsdir`)

```

Notice that you’ll be overwriting `result` in each iteration. That should ensure that Julia’s garbage collector frees the memory as you proceed.

Update: My colleague pointed out that the upload limit currently is 1GB. Soon to be increased to 5GB. This might be a limitation for you if your output is so large as your expect.

---

<div class="post-metadata">

**Author:** ![mbauman](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pumas.ai/mbauman/32/271_2.png) [@mbauman](https://discourse.pumas.ai/u/mbauman)\
**Post date:** [March 18, 2022, 4:02pm UTC](https://discourse.pumas.ai/t/jh-plugin-question/652/3 "2022-03-18T16:02:13Z")

</div>

If you tag a _directory_ as your `RESULTS_FILE`, JuliaHub will automatically do that `tar` step for you and bundle up everything that you put into that directory. Best of all, it’ll even grab intermediate results in the case of an error that crops up along the way.

Simply do:

```auto
resultsdir = joinpath(tempdir(), "results")
mkpath(resultsdir)
ENV["RESULTS_FILE"] = resultsdir

for i in 1:1000
    result = ...do stuff...

    open(join(tempname(parent=resultsdir), ".jls"), "w") do f
        serialize(f, ...)
    end
end

```

---

<div class="post-metadata">

**Author:** ![donaldlee3](https://yyz2.discourse-cdn.com/flex030/user_avatar/discourse.pumas.ai/donaldlee3/32/165_2.png) [@donaldlee3](https://discourse.pumas.ai/u/donaldlee3)\
**Post date:** [March 22, 2022, 6:05pm UTC](https://discourse.pumas.ai/t/jh-plugin-question/652/4 "2022-03-22T18:05:29Z")

</div>

Thank you both for the helpful replies. This has gone on the back burner for a bit. I’ll test it shortly and get back with any updates. Thank you again!