JH Plugin Question

Usage
1

@andreasnoack I wanted to use the plugin for a large job that would require TB’s of memory to run in one iteration. I am planning on looping through the process like 1000x and saving what I need each time through the loop. Is this possible?

The following is code is what I had to save on the plugin before. How would I modify it to save at the end of each time through a loop?

open("results_500_relF_T_3_1.jls", "w") do f
    serialize(f, Dict("MAPE_relF_summaries" => MAPE_relF_summaries, "MAPE_Vc_summaries" => MAPE_Vc_summaries, "MAPE_CL_summaries" => MAPE_CL_summaries,
                        "MPE_relF_summaries" => MPE_relF_summaries, "MPE_Vc_summaries" => MPE_Vc_summaries, "MPE_CL_summaries" => MPE_CL_summaries,
                        "MAPE_BE_cmax_summaries" => MAPE_BE_cmax_summaries, "MAPE_BE_auc_summaries" => MAPE_BE_auc_summaries,
                        "MPE_BE_cmax_summaries" => MPE_BE_cmax_summaries, "MPE_BE_auc_summaries" => MPE_BE_auc_summaries,
                        "BE_power_df" => BE_power_df))
end

ENV["RESULTS_FILE"] = "results_500_relF_T_3_1.jls"

Also, is there an efficient to manage the memory through each loop? I’m used to being able to easily clear the workspace in MATLAB. Should I do variable = nothing for all the large memory objects at the end of each loop?

And, should this outer loop be distributed with @everywhere as well?

Please let me know if there might be any other important considerations with this approach that I haven’t mentioned.

Thank you.

2

You should be able to organize your script as a loop where you save a result file in each iteration of the loop to a temporary location on the server and then, at the end, you tar up the results and mark the file for upload. I.e. something like this (warning: untested)

tarname = join(tempname(), ".tar.gz")
resultsdir = joinpath(tempdir(), "results")
mkpath(resultsdir)

ENV["RESULTS_FILE"] = tarname

for i in 1:1000
    result =  ...do stuff...

    open(join(tempname(parent=resultsdir), ".jls"), "w") do f
        serialize(f, ...)
    end
end

run(`tar -czvf $tarname $resultsdir`)

Notice that you’ll be overwriting result in each iteration. That should ensure that Julia’s garbage collector frees the memory as you proceed.

Update: My colleague pointed out that the upload limit currently is 1GB. Soon to be increased to 5GB. This might be a limitation for you if your output is so large as your expect.

2 Likes
3

If you tag a directory as your RESULTS_FILE, JuliaHub will automatically do that tar step for you and bundle up everything that you put into that directory. Best of all, it’ll even grab intermediate results in the case of an error that crops up along the way.

Simply do:

resultsdir = joinpath(tempdir(), "results")
mkpath(resultsdir)
ENV["RESULTS_FILE"] = resultsdir

for i in 1:1000
    result =  ...do stuff...

    open(join(tempname(parent=resultsdir), ".jls"), "w") do f
        serialize(f, ...)
    end
end
1 Like
4

Thank you both for the helpful replies. This has gone on the back burner for a bit. I’ll test it shortly and get back with any updates. Thank you again!