BT crash after 30 minutes+ with "module was unable to store the child's data"

[ArviLind]

Hi,

First of all thank you for this tool! As a noob i can appreciate your work and quite obviosly i dont even understand what it does so now im here.

Im using the windows open source verison of the backtester. First I deleted everything (when i couldnt fix the problem). Then did a clean windows install of gekko. And then installed the perl and followed your instructions to install the backtest tool. Actually i installed your development repo just today and that did not fix the issue. I run the BT in the cmd.exe (and have tried also it on admin/not admin). I've been running the backtests hundreds of times. It works at first and i've managed to get some smaller batched done completely (5000 combinations). And yes. im using "brute force" backtesting to find some setting to work with. But if i set a larger set of parameters in the TOML file i get i.e. 150 000 combinations that take a while to complete.

The problem is that after 20-60 mintutes of successful testing the tester runs into an error and just stops and i cant figure out why. I mean it does thousands of rounds of calculations before this so it seems extremely weird that this occurs this late.

The errors i get are below. I think they are some how connected:

USDT:NEO RSI_BULL_BEAR 50|60|30|60|20|150|20|10 -76.61% -17.14 1.48 42.86% 5.55% -16.40% 487 95 00:04
USDT:NEO RSI_BULL_BEAR 80|50|10|30|40|250|50|20 -2.52% 56.94 0.01 % % % 0 95 00:04
2018-06-08 16:57:02 (WARN): TALIB indicators could not be loaded, they will be unavailable.
USDT:NEO RSI_BULL_BEAR 50|50|10|20|40|200|50|20 3.17% 62.64 0.14 66.67% 5.16% -1.18% 310 95 00:03
2018-06-08 16:57:03 (WARN): TALIB indicators could not be loaded, they will be unavailable.

Error in tempfile() using template tmp\tmp_configXXXXX.js: Have exceeded the maximum number of attempts (1000) to open temp file/dir at backtest.pl line 1376.

2018-06-08 16:57:04 (WARN): TALIB indicators could not be loaded, they will be unavailable.
USDT:NEO RSI_BULL_BEAR 70|40|20|50|30|200|50|40 -27.91% 31.55 0.43 30.00% 0.95% -4.94% 61 95 00:04
USDT:NEO RSI_BULL_BEAR 50|80|10|50|10|300|60|30 -1.69% 57.77 0.01 % % % 0 95 00:04

Can't use an undefined value as a HASH reference at backtest.pl line 786.

USDT:NEO RSI_BULL_BEAR 70|70|30|60|40|200|40|30 -68.60% -9.14 0.09 0.00% -6.36% -52.97% 24878 95 00:04

The storable module was unable to store the child's data structure to the temp file "C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--6632.txt": can't create C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--6632.txt: No such file or directory at C:/Strawberry/perl/site/lib/Parallel/ForkManager.pm line 84.

USDT:NEO RSI_BULL_BEAR 90|80|30|30|10|300|30|10 -58.22% 1.24 0.01 % % % 0 95 00:04
The storable module was unable to store the child's data structure to the temp file "C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--7616.txt": can't create C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--7616.txt: No such file or directory at C:/Strawberry/perl/site/lib/Parallel/ForkManager.pm line 84.
USDT:NEO RSI_BULL_BEAR 70|80|10|50|40|300|20|40 -45.18% 14.28 0.20 66.67% 22.44% -47.90% 11997 95 00:03
The storable module was unable to store the child's data structure to the temp file "C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--6720.txt": can't create C:\Users\Sander\AppData\Local\Temp\O8uV2NY9VL\Parallel-ForkManager-3240--6720.txt: No such file or directory at C:/Strawberry/perl/site/lib/Parallel/ForkManager.pm line 84.

C:\Users\xxxxx\gekko>

And my config file looks like this:

no warnings qw(uninitialized);
############################# START OF CONFIGURATION #############################
Put your strategy names between brackets in line below. Strategy seperate with space or newline. You can add all Your strategies from gekko/strategies directory with adding line ALL.
@strategies = qw(
RSI_BULL_BEAR
);
Put your pairs between brackets in line below. Use exchange:currency:asset format. Seperate pair using space or newline. You can add all Your paris with ALL line or all pairs for exchange with binance:ALL line or binance:USDT:ALL line. Another option is adding dates for dataset for indivual pairs, ex: binance:BNB:NULS:2018-04-05:2018-05-01
@pairs = qw(
binance:USDT:NEO
);

Put your candle values between brackets in line below. Use CandleSize:WarmupPeriod format. Seperate pair using space or newline.
@WarmUp = qw(
5:300
);

############################# OPTIONAL SETTINGS #############################
To specify time range for import or backtest uncomment lines below, but instead this you can use command line input ex.: backtest.pl --from "2018-01-01 00:00:00" --to "2018-01-05 00:00:00". If below lines are commented Gekko is using scan datasets feature in backtest mode.
$from = '2018-03-01 00:00:00';
$to = '2018-06-05 00:00:00';

If You are using only one exchange or one exchange and one currency You can put default values below, and adding only asset name to @pairs ex: NULS, ADA, TRX - without binance:BTC before asset ex: perl backtest.pl -p NULS,ADA,TRX.
#$default_set = 'binance:BNB';

CSV file name. You don't need change this. All new data will append to exist file without deleting or replacing.
$csv = 'database.csv';

You can add note to project below. Note will be add in CSV file. Its can be useful when You are developing strategy.
$note = '8_6_dev';

Sort strategies in top list by. Available values to sort: best, profitable, profit_above_market, best_PL, worst_PL, profits_sum, avg_profit, trades_win, trades_day, hodl_time
$top_strategy_sort1 = 'best';
$top_strategy_sort2 = 'profitable';

Sort datasets ranking by. Available values to sort: best, profitable, profit_above_market, market_change, best_PL, worst_PL, profits_sum, avg_profit, trades_win, trades_day, hodl_time, price_volatility, cmc_rank, cmc_marketcap, cmc_volume, days
$top_dataset_sort1 = 'best';
$top_dataset_sort2 = 'profitable';

Template of CSV output columns. Format [% variable_name %], columns MUST be seperated by comma (,) without any space.
Below is compact version
$csv_columns = \ "[% currency %],[% asset %],[% strategy %],[% profit %],[% profit_market %],[% profit_day %],[% market_change %],[% trades_day %],[% percentage_wins %],[% best_win %],[% median_wins %],[% worst_loss %],[% median_losses %],[% avg_exposed_duration %],[% candle_size %],[% warmup_period %],[% dataset_days %],[% CMC_Rank %],[% current_marketcap %],[% open_price %],[% close_price %],[% lowest_price %],[% highest_price %],[% avg_price %],[% price_volality %],[% volume_day %],[% volume_CMC %],[% overall_trades_day %],[% dataset_from %],[% dataset_to %],[% strategy_settings %],[% note %]";
Minimalistic version - tables will dont generate
$csv_columns = \ "[% currency %],[% asset %],[% strategy %],[% profit %],[% trades_day %],[% percentage_wins %],[% best_win %],[% worst_loss %],[% avg_exposed_duration %],[% median_wins %],[% median_losses %],[% dataset_from %],[% dataset_to %],[% strategy_settings %],[% profit_day %],[% profit_market %],[% avg_price %],[% price_volality %],[% volume_day %],[% volume_CMC %],[% CMC_Rank %],[% current_marketcap %],[% overall_trades_day %],[% dataset_days %],[% market_change %]";
Full version - all possible BacktestTool variables.
$csv_columns = \ "[% currency %],[% asset %],[% exchange %],[% strategy %],[% profit %],[% profit_day %],[% profit_year %],[% sharpe_ratio %],[% market_change %],[% profit_market %],[% trades %],[% trades_day %],[% winning_trades %],[% lost_trades %],[% percentage_wins %],[% best_win %],[% median_wins %],[% worst_loss %],[% median_losses %],[% avg_exposed_duration %],[% candle_size %],[% warmup_period %],[% dataset_days %],[% backtest_start %],[% dataset_from %],[% dataset_to %],[% CMC_Rank %],[% current_marketcap %],[% open_price %],[% close_price %],[% lowest_price %],[% highest_price %],[% avg_price %],[% price_volality %],[% volume %],[% volume_day %],[% volume_CMC %],[% overall_trades %],[% overall_trades_day %],[% note %]";

Do You want coinmarketcap.com data in CSV output?
$cmc_data = 'no';

Single backtest results interval for print backtests summary (% complete, eta, avg backtest time, elapsed) for all results from current instance
$summary_interval = 100;
Print above tables on each summary?
$print_analysis_on_summary = 'yes';

Print ALL RESULTS table when use command backtest -a CSV file and at end of all backtests?
$print_all_results = 'no';
Minimum and maximum values to appear in the table with all results. Results that do not meet the following values will not be printed.
$all_results_min_profit = -999999999;
$all_results_min_profit_market = -99999999999;
$all_results_min_profit_day = -9999;
$all_results_min_trades_day = 0;
$all_results_max_trades_day = 99999;
$all_results_min_hodl_time = 1;
$all_results_max_hodl_time = 99999;
How many rows from ALL RESULTS would You print in terminal output?
$all_results_limit = 10;

Print TOP STRATEGY table when use command backtest -a CSV file and at end of all backtests?
$print_top_strategy = 'no';
Sort strategies in top list by. Available values to sort: best, profitable, profit_above_market, best_PL, worst_PL, profits_sum, avg_profit, trades_win, trades_day, hodl_time
$top_strategy_sort1 = 'trades_win';
$top_strategy_sort2 = 'avg_profit';
Limit for printed results in TOP STRATEGY table
$top_strategy_limit = 10;

Print TOP DATASET table when use command backtest -a CSV file and at end of all backtests?
$print_top_dataset = 'no';
Sort datasets ranking by. Available values to sort: best, profitable, profit_above_market, market_change, best_PL, worst_PL, profits_sum, avg_profit, trades_win, trades_day, hodl_time, price_volatility, cmc_rank, cmc_marketcap, cmc_volume, days
$top_dataset_sort1 = 'trades_win';
$top_dataset_sort2 = 'best_PL';
Limit for printed results in TOP DATASET table
$top_dataset_limit = 10;

Do you want see roundtrips report in terminal output?
$print_roundtrips = 'no';

Use TOML strat's config files instead JSON?
$use_toml_files = 'yes';
$toml_directory = 'config/strategies/';

Do you need Gekko's log files in log directory?
$keep_logs = 'no';

Threads amount, for 4xcpu cores is recommended to set 5-6 value.
$threads = 6;

When you set stfu to 'yes' only results will be displayed.
$stfu = 'no';

I've tried googeling this issue and errors above and searched the issues and suggested fixes here but i keep on getting the same errors.
Plz halp!

[AdrianoImperador]

After how many backtests it stops?
E.g. you got 1 million possible settings, but the backtester stops after a while. After how many runs it stops? Look at the CSV and let me know, maybe we got the same issue, since my backtester always stops after a batch of 1000 backtests.
Anyone experiencing the same issue?

[skincat]

I have the same problem. After 1000 backtest it stops

[ghost]

I have the same too. Happens usually hwne there is more than 500-800 tests. Happens all the time when there is more than 800

BTC:LEND mounirs-ga-version-2 -30.13% 14.09 0.28 37.50% 7.96% -15.15% 5.362.5 60:15 60
The storable module was unable to store the child's data structure to the temp file "/var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94314.txt": can't create /var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94314.txt: No such file or directory at /Library/Perl/5.18/Parallel/ForkManager.pm line 174.
BTC:LOOM mounirs-ga-version-2 -7.01% 28.75 0.27 50.00% 9.27% -22.65% 5.752.5 60:15 60
The storable module was unable to store the child's data structure to the temp file "/var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94322.txt": can't create /var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94322.txt: No such file or directory at /Library/Perl/5.18/Parallel/ForkManager.pm line 174.
BTC:LRC mounirs-ga-version-2 -29.09% 13.99 0.20 50.00% 6.10% -34.73% 6.960 60:15 60
The storable module was unable to store the child's data structure to the temp file "/var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94325.txt": can't create /var/folders/vj/0hwzqf6n4pxg303v31bjhyyr0000gp/T/vBdmwHC8Ss/Parallel-ForkManager-81041-94325.txt: No such file or directory at /Library/Perl/5.18/Parallel/ForkManager.pm line 174.

@xFFFFF how can we fix this?