Run a Neural Network first with Matt’s original experiment and mark any different or similar results (NNs can provide different results on different runs or setups). My goal is to replicate Matt’s original ‘published’ results, which were apparently positive forecasting abilities.

Then I want to modify the networks topology, specifically the number of hidden layers, and see how these changes in topology change results on the same data. If I have the time or inclination, I could try the same thing but changing the training/learning/initialization algorithms.

Because of reasons that will become apparent while reading the results, I had to modify my goal. My goal became to replicate and verify the full models operation and results with the same data that was originally used. This notably included passing forecast results into a portfolio optimization algorithm. Again, I wanted to replicate Matt’s original final results.

Increasing the number of layers in the neural net of Matt’s experiment will increase the returns (or accuracy) of Matt’s original results. This entails, first, a reproduction of Matt’s conditions in Matlab to produce the same results that Matt had. I Hope to be able to repeat and verify all of Matt’s final results.

We are attempting to gain a system that can provide stock market forecasts as to which companies out of many, will outperform the market as a whole (index’s). Forecasts are attained with simple neural networks, passing in 8 quarterly fundamental data attributes (balance sheet) as input, and quarterly market-price excess-returns as output (that is, the excess return over or under the index’s return). We pass the forecasted excess-returns into sharpes GMQP portfolio optimization algorithm to determine a portfolio of companies that may be best to invest in with a given risk acceptability.

MSE (Mean Squared Error) in the case of my experimentation on different Matlab neural network topologies was unclear to me before. MSE in this case is, how well the neural network ‘maps’ the training (and testing?) inputs to their accompanied outputs. In this specific case, we are mapping various fundamental data elements (inputs) to the next periods excess stock return (output). Excess return is how much extra return a stock showed over a common market index’s return of the same period. Excess return = Individual Stock Return - Market Index return. This allows us to use the neural net to forecast excess returns, given fundamental data.

With the above forecast, you could then implement a stock trading strategy to invest in those stocks, at the same time as shorting the market index. This would provide the investor with shielding from unknown outside world events. You would consistently gain the difference of your portfolio return over the market index’s.

Yet another strategy could involve figuring which kind of market we are in (up, down, sideways), and trade accordingly. If in a up market, you buy only the overproducing sharpe optimized portfolio. If in a down or sideways market, you revert to the aforementioned method, to get the difference of your portfolio return over the index’s return. This strategy assumes that we know what kind of market we are in, which may be another problem in itself.

gmqp.m (Gradient Method Quadratic Programming) is a file used in Matt’s methods. It comes from William F. Sharpes website at http://www.stanford.edu/~wfsharpe/mia/opt/mia_opt1.htm . It is a common portfolio optimization solution, and Sharpe provides the MATLAB code as his primary example of its implementation.

The function takes the following inputs and outputs x, the optimized portfolio:

function x = gmqp(rt,e,C,lb,ub,x0);
%  determines solution to a standard optimization problem
%  usage:
%    x = gmqp(rt,e,C,lb,ub,x0);
%      rt =  Investor risk tolerance
%      e  = {n*1} vector of asset expected returns
%      C  = {n*n} return covariance matrix
%      lb = {n*1} vector of asset lower bounds
%      ub = {n*1} vector of asset upper bounds
%      x0 = {n*1} vector of initial feasible asset mix

This script is from Matt’s original project, except that I added comments for my own complete understanding and easier following by someone unknowing of what the commands do.

The script basically loads some pre-existing input and output data, initializes and loads the data into a new neural network, trains the network with the given data, and checks errors via MSE.

% Initialize
% sets up the neural network
loaddata
net = newff(P',T',100);
% Train
net = train(net,P',T');
% Check error on training set
Y= sim(net,P');
mean((Y' - T).^2)

% Re-initialize and re-train
% Done in-case first go around's random startings was not quite optimal
% each run is different
net = init(net);
net = train(net,P',T');
% Check error of re-train
Y= sim(net,P');
mean((Y' - T).^2)

% Backtest
BACKTEST_DEC_05 = sim(net, DEC05');
BACKTEST_MAR_06 = sim(net, MAR06');
BACKTEST_JUN_06 = sim(net, JUN06');
BACKTEST_SEP_06 = sim(net, SEP06');
BACKTEST_DEC_06 = sim(net, DEC06');
BACKTEST_MAR_07 = sim(net, MAR07');
BACKTEST_JUN_07 = sim(net, JUN07');
BACKTEST_SEP_07 = sim(net, SEP07');

This script was recreated by Nick, from Matt’s notes and communications, research into gmqp and covariance, and trial and error. The detailed history of how this script was obtained is located here: run_matlab_neural_networks-test_trial_log

IMPORTANT: Read all comments when running below code. Much of the code will need to be altered dependent on circumstances of the particular run.

% Get Expected return vector
% This was forecast by the Neural Network in the prior script
E = BACKTEST_DEC_05;

% Prepare Covariance data (Historical excess relative returns)
% You want to prepare, in Excel, data that has Companys-used as the columns, and historical-quarters as the rows
% So for example, the entry in the first row, second column, is the excess return of company 2 in the first quarter of past data.
% Copy data from the 'MATLAB format data' worksheet from the 'Excess Relative Return' column
% Note: this is NOT the covariance matrix that you will feed the GMQP algorithm, but it is a pre-cursor

% Get Covariance data
% Copy/paste ‘Covariance Data’ worksheet into MATLAB as array C
% Note: Below command is not the actual command; You will need to copy/paste the full excel data straight into command line
C = [0.043	-0.084	-0.050 etc ...
];

% Check Covariance data
% IMPORTANT: Assure that the columns of the covariance data match the companies you are planning to use for the given prediction quarter
% Matt predicted 7 quarters, and each quarter had slightly differing companies used.
% Remove columns of covariance data if needed with the following command. Replace 58 with the column number to be removed.
C(:,58) = [];

% Calculate Covariance Matrix
% This will result in a n-by-n covariance matrix
C = cov(C);

% Create other needed variables for GMQP
% n  = number of companies (rows)
% UB = upper bound as a vector of ones, meaning no stock could have more than the entire amount of money available for investing (i.e., no borrowing)
% LB = lower bound as a vector of zeros, meaning no stock could be sold short
% x0 = initial allocation as an equal allocation among all stocks
n = 63;
UB = ones(n,1);
LB = zeros(n,1);
x0 = (1/n)*ones(n,1);

% Run GMQP (Gradient Method Quadratic Programming)
% The output provides you the optimally weighted companies to compose a portfolio during that quarter, per the inputs you provided
% .0001 = risk tolerance; this number happens to give a reasonable number of companies in response
OPT_WEIGHT_DEC_05 = gmqp(.0001, E, C, LB, UB, x0);

I am working toward replicating the full method that Matt exemplified. By “full” method, I mean obtaining then using the Neural Network forecast outputs (Expected Excess Return / EER) to plug into Sharpes portfolio optimization algorithm, then calculating backtested results for performance. I hoped to replicate or approximate Matt’s performance results.

The following results refers to data in the following excel document. Mostly, in the worksheet “Results”. 1-results_prototype_replication.xlsx

I was able to replicate large quantities of Matt’s results, at limited levels and with his seed data that was provided originally. I could not get the neural networks to forecast similar results, which is very disconcerting. Once I skipped that step, and used Matts original forecast data, I was able to get optimal portfolios to agree exactly (down to 10+ decimal places!). This showed me that the optimal portfolio algorithm and method seem to be solid and repeatable. The neural network portion not matching in a pair-wise comparison of the many company forecasts that it gave, is something I have not come to grips with understanding, nor have I given up on it. It continues to be an unknown variable in the answer to my aforementioned experiment goal.

Despite roughly verifying the results, much is left outstanding including:

• Neural Networks not yet giving repeatable or similar results on forecasts. This could point toward a lucky, happen-stance, or otherwise unreliable set of favorable results. This issue is not at all dead. Many avenues of thought as to why this happened and what can be done are explored below and on other pages.
• I did not tackle using differing quarters, or verifying / importing my own quarterly data. This means that it is possible (though doubtful) that the data Matt originally obtained, if erroneous or used incorrectly, could still possibly be at the root of overly exciting results.
• I have not at all touched any possibility of extending this method as a general solution to a variety of data. The possibility still exists, that the favorable results could be lucky or happen-stance, specifically for this data set chosen.
• A small amount of other intermediate data that Matt had calculated in ways that I do not understand (pre-covariance data including S&P500 return data, and excess return calculations ) was not verified. This again opens up the possibility that any results shown, are not truly repeated from scratch, and therefore not completely verified.

Below is a very quick and initial result of simple Mean Squared Error’s (explained before) on three runs of different Neural Network topologies. I have since come to find, that the MSE measured below, was the MSE of training data only, which does not tell us much. We want to know how good a network performs compared to test (backtest) data. These initial results, admittedly do not mean much, which I found out long after recording them.

Added these lines of code to make the “original run” network, a 3-layer network:

net.numLayers = 3;
net.biasConnect = [1; 1; 1;];
net.layerConnect = [0 0 0; 1 0 0; 0 1 0];   
net.outputConnect = [0 0 1];

I next attempted to step closer toward a complete replication of Matt’s project results, as close I could.

Multiple runs of the neural networks in the same setup as Matts gave EER (Expected Excess Return) unsimilar to Matts original EER. I setup the networks and data exactly the same as what Matt had documented. I calculated average absolute difference of matts values versus mine. Though not too different on average, a visual pair-wise comparison said otherwise. Why are they SO different? Another run also showed similar differences in pairwise comparison, even to the just-ran exact same setup.

The full results from one quarter are presented in the above excel file, under the column “EER”. Matts original results are listed under “Matts Orig EER”. Various runs of different neural nets are not shown, but were run. What is shown is the results of one of the runs by me. More quarters were not yet attempted because of the discrepancies I saw and noted.

I am thinking that these differences in results might be reflective of Matt’s own similar troubles that he voice to me. Matt talked to me about differing results of his own on this and similar neural net projects, even from one run to another and even with keeping the setup exactly the same. He said that he began utilizing a new method of training and comparison that he calls “leave-one-out” to solve the problem. I may need to focus on learning and using leave-one-out in order to compare neural net runs of the same data, from run to run. I do not believe that results from one run to another should differ drastically. Otherwise, forecasts will likely differ too much run to run on the same data and setups. Difference of forecasts on runs of the same data and setups, implies you are not getting reliable, repeatable forecasts.

There are large possible caveats with the leave-one-out method, so I am not sure how much time to spend on it. Explanation, Pseudo-code, and caveats were well researched and documented here: http://nickyeates.com/personal/school/financialitsystems_is698/neural_net_evol_alg_project/documentation/leave_one_out

After the above failure of replicating the neural network results, I decided that I may still be able to replicate the rest of the results, using Matts Neural Network forecast numbers, instead of mine which did not match. I needed to feed forecast numbers into portfolio optimization, and so I did this with Matt’s Expected Excess Return (EER) numbers from his submitted excel document. These are in the “Matt’s Orig EER” column’s in the “Results” worksheet in the excel document above.

I could not just simply use these numbers and paste them into somewhere. Different data, came from many undocumented places, and many undocumented calculations were done. I had to converse with Matt many times, and go over the data, enhance the data, enhance the spreadsheet for readability, and generally massage equations and workings until I got it all to work how Matt had. This process of figuring out portfolio optimization, documenting it, and scripting it took the most time to implement over anything else on this page. See the in depth log of how I did this over many days: log_of_understanding

I initially got the first quarter of results to match! I got the DEC-05 results to match exactly! Sharpes GMQP spit out the exact same results as matt listed. See this in the multiple column’s labeled “Opt. Weight Replication” in the “Results” worksheet of the above excel document. I later got most quarters results to match! I calculated an optimum portfolio for all 7 quarters with Matts data. All data matched, except for JUN-07. I think JUN-07 had an extra company in it that Matt had formerly said he removed. This likely threw off matching of the EER and Covariance’s data. You can see the erroneous results, and the row (company) likely liable for the mis-calculation highlighted bright yellow under the JUN-07 section in the “Results” worksheet.

1. Data completeness
   • Because you ussually want to run the portfolio optimization for many different quarters, the varying data completeness was a huge issue.
   • Missing data caused taking in and out data, that I did manually. It added confusion and gobs of time.
   • Varying companies with complete data, on successive quarters, caused automation efforts to be reduced to comments in the script, instead of something that could be fully automated.
   • Something has to be done about these data issues, if fuller automation is wanted.
2. Division of training, testing, and validation sets were misunderstood and mishandled
   • Matt admitted this, and said it can be done better, handled more by Matlab’s neural network scheme.
   • Default is a 60% training / 20% validation/ 20% backtesting; these can be tweaked to different percentages adding to 100%
   • The leave_one_out method completely changes training, cross-val and backtesting methods. See the link for details.
3. FIXED The excel calculation column for Even-Weighted Excess Return seems to be incorrect
   • It is being calculated as: Expected Excess Return * Even Weight
   • I think it should be calculated by Even Weight * Actual Excess Return of Stock
   • Even-Weighted Excess Return‘s counter-part: Weighted Excess Return is calculated as Optimal Weight * Actual Excess Return of Stock
   • As a reminder, in this experiment an even-weighted portfolio of all stocks, is being compared to a expected/optimal-weighted portfolio of select stocks (based off the neural network and Sharpe optimization)
4. FIXED MOST Mislabeled or misunderstood (by me) column headers in worksheet MATLAB format data:
   • The column header Next Year Relative Change in Price Close is confusing and not explained. Everything else is in quarters and months.
   • Column to the left of Excess Relative Return is not labeled. In the Workbook explanation, it mentions Return. I think this missing column may be Return.
5. FIXED MOST Many simple cells do not contain equations; they do not look to have been calculated by Excel. I think Matt did some intermediary calculations in Matlab, that he did not outright specify in his paper or elsewhere that I could find.
   • This initially did not allow me to completely finish his experiment through Portfolio Optimization and its performance.
6. FIXED Portfolio optimization. I had no idea how complex and complicated this was.
   • I spent a majority of my hours figuring out how it worked, what data it needed, and creating a script with full comments.

For Nicks note mostly. Useful Neural Net Matlab commands.

net % shows components of the object
net.trainParam.show= 1; % controlls how verbose training is
[m,b,r] = postreg(Y,T'); % FIXME

[net, tr.tperf] = sim(net, P,T) % this allows you to record test data MSE; instead of just net = sim(net,P,T) which shows training MSE
tr.tperf % last entry in this vector is test data MSE

I have a few choices. Below is a list of different questions or goals that could be hypothesized and tested in the future.

1. Does ‘leave-one-out’ work?
   1. Implement and use leave-one-out
   2. How might leave-one-out be modified to work with dynamic (time-series based) neural networks?
      1. If you take out any sample of data during the leave-one-out process, you are disjointing the time-series
2. Do ‘convexification’ methods give optimized neural net results?
   1. Do the solution-spaces successfully become more convex? Do they give more pronounced global minimas?
   2. Will any method handle high-dimension solution-spaces?
3. Do my own NN setups give good portfolios?
   1. Use own NN runs to feed data to portfolio optimization
   2. Setup environment to use different data sections and lengths (older, newer, longer, shorter)
   3. Implement automation of entire process; starting with portfolio optimization
   4. Determine and define measure of success better. Once optimal portfolio is determined, what defines that it is better than others or that it is good?
4. Does using Dynamic (time-series) based neural networks, enhance forecast results?
   1. Understand, and implement dynamic neural networks
   2. Figure out what kind of training method is used, and how validation is done
5. Does implementing a genetic algorithm to programatically change aspects of the neural networks topology (settings), enhance results?
   1. Completely automate and variable-ize neural net creation and portfolio optimization scripts in MATLAB
   2. Program Excel with VBA to automatically run MATLAB scripts and programming