Quant Basics 5: Parameter Sweep
In the last section we ran a single backtest. However, for our strategy to work we should optimise our strategy parameters. If we blindly run through a large set of parameters and then pick the best one we are very likely to fall for an issue called Data Mining Bias. This means that if we test our strategy on past data we are very likely to find a parameter set that specifically works for this particular data set but it is very unlikely that this set of parameters will work in the future. In our case, we run the parameter sweep to do something different called System Parameter Permutation (SPP). This will be explained in more detail later on. In a nutshell, for our strategy we take into account all parameter sets, not just the best one.
There are a number of ways to run parameter sweeps. The most obvious is to run through all parameter combinations one-by-one. With two parameters, this would look like a grid, with three parameters it would look like a cube and with more than three we call it a hyper-cube. The problem with this approach is that it can take a long time and, if we were to terminate our sweep early, we only end up covering a small sub-section of the total parameter space.
A better way to do this is to use Monte-Carlo methods. Rather than building a rigid grid we choose our parameters randomly. This way, our parameter space is covered more uniformly (although we will find some more dense areas) even if we terminate early. Furthermore, periodic grids can in some circumstances be bad news as they might obfuscate effects with a similar period. The figure below shows how the parameter space is covered with 500 runs. A grid sweep would have only covered a small subsection of this space.
How the parameter sweep works
The code below shows how to perform a parameter sweep. The process is rather simple.
- Pass an array with all our parameters to the function and loop through all the parameter pairs
- Sort the parameters so that the smaller is in one position and the larger in the other (to keep the strategy consistent)
- If both parameters are equal, jump to the next set (as they will not produce a crossover)
- Calculate the signals for each strategy as well as Sharpe and drawdown and save to a list
- Handle exceptions (since data can have errors)
This function will return lists of metrics where each position in the metric list corresponds to a position and the parameter array.
def parameter_sweep(tickers,p,params,N): pnls =  sharpes =  ddwns =  for i in range(N): a = min(params[i]) b = max(params[i]) if a == b: continue try: sig = calc_signals(tickers,p,a,b) pnl = calc_pnl(sig,p) pnls.append(pnl[-1]) sharpes.append(calc_sharpe(pnl)) ddwns.append(calc_ddwn(pnl)) except: pnls.append(np.nan) sharpes.append(np.nan) ddwns.append(np.nan) return pnls,sharpes,ddwns
Running the parameter sweep
In order to run the above function some more things are required. First, we need a parameter array:
np.array([np.random.randint(sm,lm,(N,)) for i in range(2)]).T
This code is called a “list comprehension” in Python. It creates pairs of random integers within a specified interval, writes them to a list and converts the list into an array.
Next, we split our strategy into two time periods which serve as train and a test set. The train set shows us the performance of our strategy during a past period and the test set shows us how the strategy would have performed out-of-sample. Traditionally, we optimise a strategy on the train set only, however, with SSP this will be different and therefore we run the test set for each parameter set as well. We need to split the dates into two periods, and the following line will determine the split point:
The timedelta function helps us to accomplish this. Finally, we can see the whole code for running a parameter sweep with train and test set below.
def run_parameter_sweep(tickers,start,end,BACKEND): N = 500 sm = 5 lm = 250 frac = 0.75 split_point = str(datetime.timedelta((parse(end)-parse(start)).days*frac)+parse(start)) print 'MID POINT:', mid_point print 'BACKEND:',BACKEND params = np.array([np.random.randint(sm,lm,(N,)) for i in range(2)]).T p0 = prices(tickers,start,split_point,backend=BACKEND) pnls1,sharpes1,ddwns1 = parameter_sweep(tickers,p0,params,N) p1 = prices(tickers,split_point,end,backend=BACKEND) pnls2,sharpes2,ddwns2 = parameter_sweep(tickers,p1,params,N) return pnls1,sharpes1,ddwns1,pnls2,sharpes2,ddwns2
Finally, we end up with an array of PnL values. In order to have a first understanding of the strategy, we plot a histogram of all the final values for each PnL series.
def plot_pnl_hist(pnls1,pnls2): plt.hist(pnls1,40) plt.hist(pnls2,40) mean1 = np.mean(pnls1) mean2 = np.mean(pnls2) plt.xlabel('pnl') plt.ylabel('N') plt.title('mean train: %s; mean test: %s'%(mean1,mean2)) plt.show()
The blue bars are the result of our train set. We can see that our strategy shows a shift towards positive PnLs, not a bad result to start with. The green bars are the results from the test set. We can see that the mean of the results is still positive. This allows us to roughly estimate the out-of-sample (OOS) shortfall we can expect from our strategy. In this case it is quite large and our OOS expectations are not fantastic. However, right now we have used completely arbitrary parameter sets and given that, the result appears to be acceptable enough to continue researching. Please note, that setting the split-point between training and test makes quite a large difference to the performance. One way to improve this is to iterate through various split-points and calculate the mean shortfall (mean-train minus mean-test) and average the results.
When we look at the shortfall in the case below we need to take into account the fact that the train period is three times as long as the test period and therefore, multiply the test PnL by three. This makes the shortfall a lot less extreme and, in fact, makes the strategy look quite reasonable as it is. Of course, we would be simply lazy if we were to stop here.
In the next section we will analyse the strategy further and introduce some basic machine learning techniques that will assist us to improve our performance.
The code base for this section can be found on Github.