Recently, I have been doing a lot of web development. I wanted to use a reasonably non-trivial C++ library of statistical functions in a web application that uses a basic client-side HTML/CSS/JavaScript stack. The intention here, as on previous occasions (Software Interoperability), was to see how feasible (and easy or otherwise) it is to build a component to connect a native C++ library to JavaScript. The more general use-case is to provide access to any native C++ functionality and make it available in a browser (hence cross-platform).

This blog post describes the steps required to compile a native C++ library to WebAssembly(WASM), and to use this in a simple web application. The resulting application is shown below.

The full project is available here.

Emscripten

The main tool that we use is emscripten. The documentation describes emscripten as a complete Open Source compiler toolchain to WebAssembly. More specifically using Emscripten you can:

• “Compile C and C++ code, or any other language that uses LLVM, into WebAssembly, and run it on the Web, Node.js, or other Wasm runtimes.”
• “Compile the C/C++ runtimes of other languages into WebAssembly, and then run code in those other languages in an indirect way (for example, this has been done for Python and Lua).”

My own interest here is in compiling a C++ library for use in a basic web application. Specifically, the goal is to produce a WebAssembly module that contains (and exports) all of the functionality from the library that we want to consume from JavaScript. emscripten provides all the necessary tools to manage this. It is also well-documented. Follow the Emscripten documentation for instructions on how to install the SDK. The rest of this post will assume that you have activated the latest Emscripten SDK in your PATH via source ./emsdk_env.sh (or emsdk_env.bat on Windows).

The C++ Library

The C++ library that I use is called StatsLib. The library is written in C++20 and uses boost 1.91. I have used this on previous occasions to demonstrate software interoperability with C++ from languages like C#, Python and R. The library consists of a number of classes and functions to perform basic statistical analyses. DescriptiveStatistics provides summary data for a given dataset. LinearRegression performs a univariate linear regression. And there are a number of classes derived from StatisticalTest that form a class hierarchy for handling statistical hypothesis tests. There are classes to perform a student’s t-test, a z-test and an F-test. There is also a class that performs a basic moving average as part of a time series analysis. Finally, the DataManager class provides a simple caching mechanism for datasets. It enables users to load datasets from files, then store and retrieve them by name on demand.

There were two main reasons for using this library. Firstly, I wanted an example that was not completely trivial as this obscures and ignores important (possibly critical) issues. But also not too complex like the SkiaSharpViews. A good number of WebAssembly examples are based on a single C function (add, fib etc.) with simple ‘blittable types’. However, I want to be able to use STL types (std::vector, std::map, std::string), and I also want to use classes, methods, and properties.

Secondly, most of the information about WebAssembly focuses specifically on the performance aspect rationale for using a C++ library in a web application hosted in a web browser. However, I think there is a more general use case applicable to any C++ code. One of the main attractions of WebAssembly is that it allows us to leverage the vast library of existing open-source C and C++ code in web applications.

Project

StatsLibScript consists of a single project.

.

The StatsLibScript folder is divided into \lib and \web sub-folders. The \lib folder contains the sources and the \web folder contains the web application.

The StatsLibScript Component

The \lib folder contains the C++ code that is compiled into wasm. This consists broadly of the emscripten bindings and auxilliary classes/functions that help with the communication between C++ and JavaScript.

The auxilliary classes/functions are:

• Conversion.h/Conversion.cpp

  This is where we locate any functions needed to convert between native C++ (and STL) types and types understood by JavaScript/Wasm. The only conversion of interest concerns the results returned from StatsLib functions. In general these are returned as a std::unordered_map<std::string, double>. However, even after registering this type (see below), wasm did not appear to ‘process’ it. As far as I could tell, there is no template defined for this MapType in bind.h. Therefore, I wrote a conversion to a std::map<std::string, double>. An earlier version of the component made use of emscripten::val v which allows converting JavaScript types to C++. However, these turned out not to be needed as the bindings specified in module.cpp (see below) worked directly.

• Functions.h/Functions.cpp

  This contains all the function wrappers which are called through the bindings.

• Classes.h/Classes.cpp

  This contains proxy classes for the DataManager, TTest and ZTest classes of StatsLib. These are not strictly necessary if the C++ types can be registered (or for classes where the specific functionality is not exposed). But with functionality like DataManager::ListDataSets we needed to convert from std::vector<DataSetInfo> to the registered ‘VectorString’ type i.e. std::vector<std::string>. Similarly, to handle the results coming back from TTest and ZTest classes we convert to the registered ‘MapStringDouble’ type i.e. std::map<std::string, double>.

• module.cpp

  This contains the main module with the emscripten bindings. The macro EMSCRIPTEN_BINDINGS(my_module) contains the definitions of the bindings. These are the functions that JavaScript will call and the classes that can be used.

In detail, first off, we register vector and map types. This allows us to use these types conveniently from JavaScript, e.g.:

let xs = new Module.VectorDouble();
xs.push_back(0.0);
xs.push_back(1.0);
...

This vector can be passed directly to C++ functions/classes. We can also iterate over the values and convert to a JavaScript array:

let arr = new Array(xs.size()).fill(0).map((_, id) => xs.get(id));
console.log("DataSet: ", arr);

The module also defines the functions we want to overload. We use the select_overload method in embind to define two overloads for the DescriptiveStatistics function, with or without a list of keys that identify the results we want to retrieve:

let keys = new Module.VectorString()
keys.push_back("Mean");
keys.push_back("StdDev.S");

const results = Module.DescriptiveStatistics(xs, keys)

console.log("Mean: ", results.get(keys.get(0)));
console.log("StdDev.S: ", results.get(keys.get(1)));

We also declare an enumerated type mapped to the underlying C++ type:

enum_<StatisticsLibrary::DescriptiveStatistics::VarianceType>("VarianceType")
    .value("Sample", StatisticsLibrary::DescriptiveStatistics::VarianceType::Sample)
    .value("Population", StatisticsLibrary::DescriptiveStatistics::VarianceType::Population);

This can then be used in JavaScript:

console.log("Standard Deviation (Population): ", Module.StandardDeviation(xs, Module.VarianceType.Population));
console.log("Standard Deviation (Sample): ", Module.StandardDeviation(xs, Module.VarianceType.Sample));

Finally, we define the wrapper classes in terms of constructors and functions. The DataManager class takes a simple constructor, followed by a number of functions. The TTest/ZTest wrappers both have multiple constructors. It is worth pointing out that the overload resolution is done using the number of arguments, not the types. This causes a slight issue in the case of the T-Test. The arguments for the second and third constructor are either a double and a std::vector<double> or two std::vector<double>’s. Since they have the same number of arguments, embind doesn’t distinguish between the two.

For example,

.constructor<double, double, double, double>()                          // 4 args - OK
.constructor<double, const std::vector<double>&>()                      // 2 args - double and vector
.constructor<const std::vector<double>&, const std::vector<double>&>()  // 2 args - vector and vector

When calling this from JavaScript, it does not distinguish the case of 2 args. To work around this we added a dummy initial third argument, which is not ideal.

Depending on the exact requirements of the C++ library that is being exposed, there is a lot more that can be done. This is covered in the emscripten documentation. This project only just scratches the surface.

Compiling & Building

Compiling and building consists of a single command line:

emcc --bind module.cpp Conversion.cpp Functions.cpp Classes.cpp 
	<path>/SoftwareInteroperability/Common/source/DataManager.cpp 
	<path>/SoftwareInteroperability/Common/source/DescriptiveStatistics.cpp 
	<path>/SoftwareInteroperability/Common/source/StatisticalTests.cpp 
	<path>/SoftwareInteroperability/Common/source/StatisticsLibrary.cpp 
	<path>/SoftwareInteroperability/Common/source/TimeSeries.cpp 
	-O0 
	-o StatsLibScript.js 
	-sENVIRONMENT="web" 
	-I D:/Libraries/boost_1_91_0/ 
	-I <path>/SoftwareInteroperability/Common/include/ 
	-std=c++20 
	-sNO_DISABLE_EXCEPTION_CATCHING 
	--no-entry

We have referenced the sources directly, just for convenience. We could equally have built (an LLVM compatible) library file and included it. Use this link for additional information regarding building projects. There are a number of useful compiler flags. We have only used a small subset here. The break down is as follows:

• -O0 optimisation level for Debug
• -o output a java script ‘bootstrap’ file
• -sENVIRONMENT=”web”
• -I …include boost library and the common includes for StatsLib
• -std=c++20 specifies the C++ language standard
• -sNO_DISABLE_EXCEPTION_CATCHING flag to indicate that we want to enable exception handling. This allows us to decorate the JavaScript functions with try { ... } catch(e) {} blocks and process the native C++ exception.
• –no-entry

After the build, we copy the artifacts to the Debug or Release directory.

copy "StatsLibScript.js" "../web/Debug/StatsLibScript.js"
copy "StatsLibScript.wasm" "../web/Debug/StatsLibScript.wasm"

For convenience, I have set this up so that it can be built from a single command line and run (using live server) from Visual Studio Code. This seemed to me the easiest approach for development. However, there is also a CMake project and a Visual Studio 2022 solution if you prefer. Further details relating to the build can be found in: Build Commands.md.

The ‘Statistics Viewer’ web application

The \web folder contains the web infrastructure and the debug and release build outputs. There is a single index.html file and a stylesheet (styles.css). index.html contains the main code to load the StatsLibScript.js module.

To use the WebAssembly module, we reference the compiled StatsLibScript.js in the script element:

<script src="Debug/StatsLibScript.js"></script>

We also need a script tag to initialise the module.

	Module.onRuntimeInitialized = _ => {

		const v = Module.getVersion();
		console.log("Version =", v);

		console.log("Initializing DataManager...");

		dataManager = new Module.DataManager();

		console.log("Number of DataSets: ", dataManager.CountDataSets());
	};

An alternative is to fetch the wasm code directly.

The rest of the html code deals with calling the functions and processing results.

The web application allows the user to load datasets and then perform statistical analyses, for example request descriptive statistics or carry out a linear regression. In terms of the UI, the web application consists of a left-hand side collapsible menu. The main page content consists of sections referenced by the navigation menu. Each section consists of a left-hand panel that contains inputs (typically dataset(s) and/or single values) and a Run>> button. The right-hand panel is used to display the outputs (typically a table, but for the Moving Average we use a graph).

To use the application, run a local server and load the index.html page. Then load some data sets into the DataManager. A dataset is a single .csv file with a column of data and a heading. The heading is used as a dataset name. There are a number of examples in the SoftwareInteroperability\Data directory.

Here we see a screenshot after four datasets have been loaded.

Then, depending on the dataset, we can choose Descriptive Statistics, Linear Regression, TTests, or Moving Average. Select the appropriate dataset(s) and press ‘Run’. The results are shown in a table on the right-hand side.

Debugging

Debugging makes use of the developer tools available in the web browser.

Conclusion

In this blog post we have seen how to take a native C++ library and compile it to WebAssembly using emscripten. We have seen how to use embind to define the types, classes and functions that we want to expose to JavaScript. And we have built this into a simple ‘Statistics Viewer’ web application.

See also

• Emscripten.
• WebAssembly binary toolkit (wabt): a suite of tools for inspecting and manipulating WebAssembly files.

In the previous posts in this series (Using R in Excel) I have demonstrated some basic use-cases where using R in Excel is useful. Specifically we have looked at descriptive statistics, linear regression, forecasting, and calling Python. In this post, I am going to look at inferential statistics and how R can be used (in Excel) to perform some typical statistical tests. Excel provides many excellent facilities for data wrangling and analysis. However, for certain types of statistical data analysis the limitations of the built-in functions and the Analysis ToolPak is not sufficient, and R provides superior facilities.

The workbook for this part of the series is: “Part V - R in Excel - Inferential Statistics.xlsx”. As before, the ‘References’ worksheet lists links to external references. The ‘Libraries’ worksheet loads additional (non-default) packages. In this demonstration, I use ggpubr and the lsr package. The ggpubr package provides some easy-to-use functions for creating and customizing ‘ggplot2’-based publication ready plots. The lsr package provides some useful wrappers for common hypothesis tests and basic data manipulation. These packages should be present and loaded before continuing with the analyses.

The ‘Datasets’ worksheet contains the data referenced in the worksheets. The data for the main analyses represents the weights (in Kg’s) of 9 women and 9 men. The data has been loaded and an Excel table created named tWeightData.

Analysis I

Creating the data frame

The first thing to do is to create an R data frame. However, rather than having a table with two columns (women_weight and men_weight) we want a ‘Group’ column and a ‘Weight’ column. To do this we first create vectors using:

=RScript.CreateVector('women_weight',Datasets!N2:N10)

and

=RScript.CreateVector('men_weight',Datasets!O2:O10)

After this, we create the data frame specifying the grouping factors and the data. The corresponding vectors and the weight_data data frame should appear in the Excel R AddIn panel.

Comparing Means

The first analysis checks whether there’s a significant difference in the weights of the different groups. For this, we perform an independent samples t-test. Prior to performing the test, it is useful to provide a summary of the data, so we group the data to obtain the means and standard deviations, and we also produce a boxplot.

From the plot, there seems to be a plausible difference in the means. It is worth noting in passing that we use the standard ggboxplot(...) function, but wrap it in a call to the plot(...) function. This outputs the result to a separate graphics window, from which we can paste the resulting graphic as a Windows metafile. This is not ideal, but as yet I have not found a way to draw a plot directly in an Excel worksheet.

Before doing the t-test we want to check the independent t-test assumptions. Specifically, the two samples are independent since the observations come from distinct groups (men and women). We also want to check that the data from each of the groups follow a normal distribution. We perform a Shapiro-Wilk normality test for each group. We could also have inspected the data visually with a histogram or a qqplot.

From the output of the Shapiro-Wilk test, the two p-values are greater than the significance level 0.05 implying that the distribution of the data are not significantly different from the normal distribution. Therefore, we can assume the observations from each group follow a normal distribution.

Comparing Variances

In addition to the Shapiro-Wilk test, we perform an F-test to compare variances since one of the assumptions of a t-test is that the population variances should be equal. The output of var.test is a list containing the individual results, which we display. The p-value of F-test is p = 0.17136. Since it is greater than the significance level (alpha = 0.05) we conclude that there is no significant difference between the variances of the two sets of data. Therefore, we can use the classic t-test which assume equality of the two variances.

Unpaired two-samples t-test

Finally, we compute an independent t-test. As before, we extract the individual statistics. The p-value of the test is 0.01327, which is less than the significance level (alpha = 0.05). We can conclude that men’s average weight is significantly different from women’s average weight in this dataset with a p-value = 0.01327.

We can also check that Excel’s =T.TEST() function produces the same result.

Analysis II

One-way ANOVA

For the second analysis, we want to perform a one-way ANOVA test. We use the built-in R data set named PlantGrowth. It contains the weight of plants obtained under a control and two different treatment conditions.

As before, we display a brief summary of the data and a boxplot to visually display the means of each group. Then we compute the one-way ANOVA. The output contains a lot of data, so we extract the individual components as required.

We can also use the anova function to summarise the test results.

As the p-value is less than the significance level 0.05, we can conclude that there are significant differences between the groups.

Because the test result is significant, we compute Tukey HSD for performing multiple pairwise-comparison between the means of groups. It can be seen from the output, that only the difference between trt2 and trt1 is significant with an adjusted p-value of 0.012. The same test can be performed using Excel’s Analysis ToolPak ‘Single Factor ANOVA’ function. This is demonstrated in the ‘Single Factor ANOVA’ worksheet.

Wrap Up

In this post we have used R in Excel to perform some basic tests for inferential statistics. We have performed a Shapiro-Wilk test, an F-test, an independent samples t-test and a one-way ANOVA. In passing, we have also seen how to plot data and how to set up and retrieve model results. While some of this could have been done using Excel’s native functions, the advantage of using R from within Excel is that it gives us access to an extensive ecosystem, which in turn gives us a good deal of flexibility in creating solutions.

The purpose of this series of posts is to demonstrate some use-cases for R in Excel using the ExcelRAddIn component (disclaimer: I am the developer of this add-in: ExcelRAddIn). The fundamental rationale for the add-in is that it allows access to the extensive R ecosystem within an Excel worksheet. Excel provides many excellent facilities for data wrangling and analysis. However, for certain types of statistical data analysis, the limitations of the built-in functions even alongside the Analysis ToolPak is not sufficient, and R provides superior facilities (for example, for performing LDA, PCA, forecasting and time series analysis to mention a few).

This series of posts demonstrates four main areas where R is useful in Excel: descriptive statistics, linear regression, forecasting, and accessing Python. Along the way, we will see that using R in Excel is no more difficult than writing a formula and calling the ExcelRAddIn to evaluate it. The ‘trick’, if there is one, is unpacking the results into a form that Excel understands and which can be used in a worksheet. We will see several examples of how to do this.

Installing and setting up the ExcelRAddIn is described here. Each part of the series is accompanied by an Excel workbook with the R scripts. The workbooks depend on the ExcelRAddIn-AddIn64.dll, so this should be loaded first, and the “R AddIn” menu should appear on the right-hand side of the menu bar.

The task pane is empty until the first script is evaluated. This will initialise R using the directories in the “Settings”. The default packages I use are tidyverse, dplyr, forecast, ggplot2, ggthemes, as shown below:

The workbook for this part of the series is: Part I - R in Excel - Descriptive Statistics.xlsx. The workbooks all have a similar structure just to keep things organised. The ‘References’ worksheet lists any links to external references. The ‘Libraries’ worksheet loads additional (non-default) packages. The ‘Datasets’ worksheet contains any data referenced in the worksheets.

Descriptive statistics

Loading Data

The first step is to load some data. This dataset comes from “Linear Models with R” by Julian Faraway. In this example, I have loaded the data into Excel from a csv file (GalapagosData.csv) using Power Query. The data has been tidied up and I have created a table (tableGalapagos) that can be referenced in the workbook.

In the Descriptive Statistics worksheet, we first create a data frame using the CreateDataFrame function.

=RScript.CreateDataFrame("galapagos", tableGalapagosData, tableGalapagosData[#Headers])

This function is part of the add-in and simplifies the creation of dataframes. There are also functions to create vectors and matrices. We pass in a name (which will appear in the R environment) and the data and headers. The final parameter (‘Type’ => character, complex, integer, logical, numeric) is optional; the RType is now determined from the data if possible. This makes it somewhat easier to create objects to pass to R from Excel.

This copies the data into the R environment. There are a number of alternatives to this approach. We could have loaded the csv file directly into R using:

galapagos <- read.csv("D:\Development\...\GalapagosData.csv")

By loading the data into Excel and then copying it to R, we can leverage the Excel import using Power Query, so we automatically get grouping, filtering etc, and can immediately create PivotTables. The downside is that we need to make a copy into R, and this means ensuring the data types are ‘viable’. This is particularly important with dates.

Obtaining Statistics

Now that we have the data frame in R (and in Excel), we can obtain some descriptive statistics. If we were doing this exclusively in Excel, we might use individual formulas (=COUNT(), =AVERAGE(), =STDEV.S() and so on). Using R, we can accomplish the same thing.

As expected, this returns the mean and standard deviation.

We can improve on this using some additional R functions: sapply together with fivenum (fivenum returns Tukey’s five number summary (minimum, lower-hinge, median, upper-hinge, maximum) for the input data).

as.data.frame(sapply(galapagos[,2:8], fivenum))

When evaluated, all this does is to apply the fivenum function to columns 2 to 8 (zero-based) of the galapagos dataset, and coerce the result into a dataframe. If we don’t do this we don’t get back the column headings (which is not tremendously useful). You may also have noticed that we have to determine the return values from the documentation. There doesn’t seem to be a way of retrieving this meta data from the function.

An alternative to the five-number summary is to use the built-in summary(...) function. Unfortunately the output from this does not work well with Excel, so we need to massage the results in order to obtain a decent table showing the summary with labels.

Basically, we get the column labels from the galapagos dataset using: names(galapagos), and we get the labels for the summary using: names(summary(galapagos$Species)). Then for each column of the data we are interested in, we request the summary. For example: summary(galapagos$Elevation).

Now that we are massaging the results, we might even consider using a custom function. We can, for example, define a function that returns a dataframe consisting of a label and the corresponding statistic:

custom_summary <- function(data) {
  label <- c("count", "mean", "std.dev")
  value <- c(length(data), mean(data), sd(data))
  data.frame(label, value)
}

Evaluating the function with the script:

custom_summary(galapagos$Area)

outputs a small table, as follows:

If all this seems quite a lot of hard work, or you are looking for a more sophisticated approach - perhaps a summary that tests the normality of the distribution of the data - then you might prefer to use a summary function from a different package. There are several to choose from. Here we use pastecs pastecs and summarytools summarytools. Both of these give good results with minimal effort:

Wrap up

In this post, we have seen various approaches to obtaining descriptive statistics using R in Excel via the ExcelRAddIn. We have introduced some basic approaches (similar to what Excel natively offers). But we have also seen some more advanced usage that demonstrates how useful it can be to have access to R functionality in Excel. The next post covers linear regression using R in Excel.

In the first part of this series, I looked at using R in Excel to obtain descriptive statistics. In this second part of the series I am going to look at using R in Excel to perform linear regression, specifically using the lm() function. lm() is a real workhorse function. It can be used to carry out both single and multiple regression and different types of analysis of variance. For this demonstration I will only focus on single and multiple regression.

The workbook for this part of the series is: “Part II - R in Excel - Linear Regression.xlsx”. As before, the ‘References’ worksheet lists links to external references. The ‘Libraries’ worksheet loads additional (non-default) packages. In this demonstration, I use the datarium and broom packages. The ‘Datasets’ worksheet contains the data referenced in the worksheets.

The Data

The dataset is the marketing dataset from the datarium package. It consists of a data frame containing the impact of three advertising medias (youtube, facebook and newspaper) on sales. Data are the advertising budget in thousands of dollars along with the sales. The advertising experiment has been repeated 200 times. The dataset is described here.

If the dataset is not available, check that the datarium library is loaded by going to the Libraries worksheet and evaluating:

=RScript.Evaluate(library(datarium), TRUE)

Simple Linear Regression

In the Simple Linear Regression worksheet we create a simple linear model.

smodel <- lm(sales ~ youtube, data = marketing)

As we did in the first part of this series, we unpack the ‘smodel’. Mostly this is simply a question of evaluating the corresponding R objects, for example the residuals and the coefficients. We also request the 95% confidence intervals for the coefficients.

At this point, we can compare the results to the Summary Output provided by the Analysis ToolPak’s regression function (including the 95% confidence intervals).

It’s worth emphasising that the output that is returned to Excel via the add-in is not ‘formatted’ as it is in R/RStudio. If we call summary(smodel) in R we are presented with a familiar tabular output summarising the main features of the model. However, what is returned to Excel via the add-in is more basic. So it is worth spending some time looking at this. We can see from the add-in environment, that the smodel_summary is a list of 11 items. To use this we need to unpack some of the individual list items.

Here we can see the actual call made and the details of the model including the statistics (sigma, r-squared and so on). It needs some extra work to unpack the summary data from the model residuals in order to produce a tabular output.

as.data.frame(as.array(summary(smodel$residuals)))

This formula summarises the model residuals and coerces the data into an array first (to get the names) and then as a data frame. We perform a similar operation to get the coefficients into a tabular format.

With all the data available in this form, and with a little effort, we can now construct a summary output similar to R.

Multiple Linear Regression

The next worksheet demonstrates multiple linear regression using the same marketing data. This is similar to the previous worksheet and illustrates how to extract relevant values from the returned model data.

However, in this case we make use of the package broom. This can help tidy up the output data.

In previous examples, we have extracted model results by concatenating the returned model name with labels from the model. To make it easier to retrieve results, the add-in provides some additional functions for querying models: Model.Results outputs a list of results from the model. Model.Result outputs the result obtained from one item of the list of model results. Optionally, the result can be formatted as a data frame. This is somewhat more convenient than having to evaluate scripts of the form model name'$coeffcients, etc. By extracting the model data and the summary, we can construct an output that is similar to that provided by R.

Logistic Regression

The final example demonstrates logistic regression. At this point you might be thinking that this massaging and extracting output data is not entirely satisfactory. It’s a lot of work and it can be quite brittle (if the cell references change, for example). Furthermore, as the models become more complicated (see Part III in the series) this approach potentially becomes messier. To mitigate this, the add-in provides a couple of wrapper functions around regression. These help with both the setup and the output of more complex models.

The model inputs are specified as a block of parameters, and the function Regression.GLM is used instead of the R script equivalent using glm. The model outputs can be ‘queried’ using the Model.Results and the Model.Result formulas. However, even with this, it takes some effort to output a summary similar to the one provided by R.

Wrap Up

In this post, we have looked at how to use R in Excel to perform linear regression, and we have spent some time demonstrating how to extract the different components from the model data. This will be useful in the next two parts of this series.

We have already seen how to obtain descriptive statistics in Part I and how to use lm() in Part II. In this part (Part III) of the series we will look at using R in Excel to perform forecasting and time series analysis.

In the previous two parts we have seen different ways to handle the output from R function calls, unpacking and massaging the data as required. In this part we are going to focus on setting up and interacting with a number of models in the ‘forecast’ package (fpp2).

The workbook for this part of the series is: “Part III - R in Excel - Forecasting.xlsx”. As before, the ‘References’ worksheet lists links to external references. The ‘Libraries’ worksheet loads additional (non-default) packages. In this demonstration, I use the fpp2 package. The ‘Datasets’ worksheet contains the data referenced in the worksheets.

Moving Average

In this worksheet we are looking at the moving average function in the forecast package. ma computes a simple moving average smoother of a given time series. The data we use consists of ice cream sales.

In the previous posts we have used ‘scripts’ to call R functions. So, in this case we could evaluate

ma(Sales, 3, FALSE)

and get back the smoothed time series.

However, as the models become more complex, and the number and types of parameters increases, this approach becomes less manageable. An alternative is to use one of the custom Forecast. functions provided by the add-in. This allows us to set up the model parameters as a block of named parameters with their corresponding values. We have already seen an example of this in Part II where we set up a logistic regression model. In the case of Forecast.MA, we give the model a descriptive name. We know from the documentation that the model takes two parameters: the order (“order of moving average smoother”) and the center (“if TRUE, then the moving average is centred for even orders”). Then we just call a regular Excel worksheet function.

The add-in provides wrappers around a number of the functions in the forecast package. These are as follows:

• Forecast.MA - Moving average smoothing
• Forecast.SES - Simple exponential smoothing
• Forecast.Holt - Holt exponential smoothing
• Forecast.HW - Holt-Winters exponential smoothing.
• Forecast.AutoETS - Exponential smoothing state space model.
• Forecast.Arima - Auto-Regressive Integrated Moving Average model
• Forecast.AutoArima - Fit best ARIMA model to univariate time series
• Forecast.FC - Generic function for forecasting from time series or time series models
• Forecast.meanf - Mean forecast
• Forecast.rwf - Forecasts and prediction intervals for a random walk with drift model
• Forecast.splinef - Local linear forecasts and prediction intervals using cubic smoothing splines
• Forecast.thetaf - Forecasts and prediction intervals for a theta method forecast
• Forecast.Croston - Forecasts and other information for Croston’s forecasts

Simple Exponential Smoothing

In this example, we create a simple exponential smoothing model. As before we set up the model parameters following the forecast package ses documentation. Fortunately, most of the values are defaulted so we only provide inputs for the following:

• alpha - the value of the smoothing parameter for the level. If NULL, it will be estimated.
• h - the number of periods for forecasting.

Then we can call the add-in function Forecast.SES(...).

The model returns the requested forecasts with the mean, and lower and upper limits for the prediction intervals.

In previous examples, we have extracted model results by concatenating the returned model name (‘SES3’ in this case) with labels from the model. As illustrated in Part II, in order to make it easier to retrieve results, the add-in provides some additional functions for querying models: Model.Results outputs a list of results from the model. Model.Result outputs the result obtained from one item of the list of model results. Optionally, the result can be formatted as a data frame. This is somewhat more convenient than having to evaluate scripts of the form model name'$coeffcients, etc. Finally, Model.Accuracy returns a number of statistics relating to measures of model accuracy.

The Holt-Winters model

The HW worksheet illustrates working with the Holt-Winters model. In this case, we set up two different models with their respective parameters, each with their own parameter block: an additive model and a multiplicative model. However, we first need to address an issue with the ‘data type’ of the dataset. Typically we can just create a numeric vector. Unfortunately, the series parameter of the Holt-Winters model requires information about the frequency with which the observations in the vector occur. Therefore we create a numeric vector first and then convert it into a ts time-series vector with embedded frequency information. Currently, the ExcelRAddIn does not support time series as a data type. This will be added as a future enhancement.

ARIMA

The final example concerns the ARIMA model. The setup of the model is somewhat more complicated than in previous cases. Specifically, the order and the seasonal parameters.

From the documentation we know that the order parameter is a specification of the non-seasonal part of the ARIMA model: the three components (p, d, q) are the AR order, the degree of differencing, and the MA order. To handle this specification, we input the components using the vector notation: c(0, 1, 1).

For the seasonal parameter, the documentation says: “A specification of the seasonal part of the ARIMA model, plus the period (which defaults to frequency(y)). This should be a list with components order and period, but a specification of just a numeric vector of length 3 will be turned into a suitable list with the specification as the order.”

To achieve this we add a script parameter. This is a parameter with a leading ‘^’ character as follows:

^list(order=c(0,1,1),period=12)

With this setup we can create the ARIMA model.

The results can be extracted using a combination of Model.Results and Model.Result functions or by using a ‘raw’ R script. The model accuracy can be checked via the function Model.Accuracy(...).

Wrap Up

In this post we have used R in Excel to perform some forecasting and time series analyses. We have seen how to set up and retrieve model data. In addition to evaluating R scripts, we have also made use of a number of wrapper functions around the Forecast package provided by the add-in to make model setup and evaluation easier.

In the last part of the series I will look at using R in Excel to call Python scripts.

In the last post in this series I am going to look at calling Python from R. Even though Excel now provides a means of calling Python scripts directly, using the =PY() formula in a worksheet, there are still occasions when it is beneficial to call Python via R. For example, it turns out that importing yfinance produces a ‘module not found’ error using Excel’s function. According to the documentation, yfinance is not one of the open source libraries that the Excel Python secure distribution supports. To get around this issue, we can use the R package Reticulate. This lets us load and run Python scripts from R. As we have seen in the previous parts of this series, the ExcelRAddIn allows us to run R scripts from an Excel worksheet. And putting these two together is quite simple.

The workbook for this part of the series is: “Part IV - R in Excel - Calling Python.xlsx”. As before, the ‘References’ worksheet lists links to external references. The ‘Libraries’ worksheet loads additional (non-default) packages. In this demonstration, I use the reticulate package and this is loaded here. The ‘Datasets’ worksheet contains the data referenced in the worksheets. In addition there are two auxilliary workksheets: Session Info and Setup. The Session Info retrieves information about the R environment and R version. The Setup worksheet performs some initialisation required for the reticulate package. First we point to the Python executable that we want to use. Next, we confirm the Python version is actually the one requested. Finally we set up a section that we can return to if there are errors from Reticulate and Python itself. It is worth ensuring that these cells are evaluated correctly.

Ticker Info

This worksheet requests some basic ticker info for a number of stocks. The first step is to load the script ticker_data.py and ensure it compiles.

The Python script

The script consists of two main functions: get_ticker_data and get_ticker_weights.

"""
ticker_data.py

Retrieve ticker data from yfinance

"""

import pandas as pd
import yfinance as yf


def get_value(info: dict, key: str):
    """
    Retrieve a value from a ticker info dictionary
    or None if the key doesn't exist
    """

    if info is not None:
        if key in info.keys():
            return info[key]

    return None


def get_items(symbol: str, fields: list) -> list:
    """
    Retrieve data from the corresponding fields for the specified ticker
    """

    items: list = [symbol]

    try:
        tk: dict = yf.Ticker(symbol)
        info: dict = tk.info

        for field in fields:
            items.append(get_value(info, field))
    # pylint: disable=broad-exception-caught
    except Exception:
        # pylint: enable=broad-exception-caught
        pass

    return items


def get_ticker_data(symbols: list, fields: list) -> pd.DataFrame:
    """
    Retrieve the specified fields from the input tickers
    """

    headers: list = ["Ticker"]
    for field in fields:
        headers.append(field)

    rows: list = []

    try:
        for symbol in symbols:
            rows.append(get_items(symbol, fields))

    # pylint: disable=broad-exception-caught
    except Exception:
        # pylint: enable=broad-exception-caught
        print("An error occured")

    df = pd.DataFrame(rows, columns=headers)

    return df


def get_ticker_weights(ticker_values: dict) -> pd.DataFrame:
    """
    Returns:
        ticker_weights
    """

    # Set up the structure
    rows: list = []
    headers: list = ["ticker", "weight"]

    # Calculate the portfolio value
    total_portfolio_value = sum(ticker_values.values())

    for ticker, value in ticker_values.items():
        weight = value / total_portfolio_value
        rows.append([ticker, weight])

    df = pd.DataFrame(rows, columns=headers)

    return df

get_ticker_data takes a list of ticker symbols and a list of fields. The fields are the items we want to retrieve from yfinance’s ticker info function. It uses the utility function get_value to retrieve the requested value or 0 if the key is not available. No doubt the Python script could be simplified and made more pythonic.

In the worksheet, we create a vector of tickers and a vector of fields, and then call the Python function.

The result is a data frame with the values for the fields that we requested (if available).

Portfolio Weights

The second function get_ticker_weights, simply computes the weights of a list of ticker symbols. The Python function requires that the input is a dictionary of symbols and their corresponding values. Therefore we create a ticker_list via R, since Reticulate converts R lists to Python dictionaries. The construction is somewhat mechanical and could certainly be improved just by changing the Python function inputs. With the ticker_list created we can call the Python function.

This returns the portfolio weights which we can use to display graphically. From here, we could extend the Python code to compute portfolio returns for the tickers over a specified period. We could then compare the returns with a benchmark, and compute Sharpe ratios for the components.

Wrap up

In this post we have seen how to call Python scripts from R using the reticulate package. While the setup is not completely trivial, it is worth bearing in mind the advantages of this approach: from within Excel, we have access to both R and Python’s extensive ecosystems, which in turn gives us a good deal of flexibility in creating solutions. And that concludes this series of posts on using R in Excel. I hope this has been useful.

Recently, I’ve been doing some portfolio analysis and I came across an interesting video: How to Calculate the Intrinsic Value of a Stock like Benjamin Graham. Since not all the data is available in Excel, I thought it might be useful to automate the calculation using data from the (excellent) yfinance Python library. The formula itself, written in Python, is quite simple. The challenge was to obtain the individual parameters and inputs and be able to call the Python function from an Excel worksheet.

It would have been nice to be able to use Excel’s Python support PY=. But it turns out that importing yfinance produces a module not found error.

.

According to the documentation yfinance is not one of the open source libraries that the Excel Python secure distribution supports.

So, in order to bridge the gap between Excel and Python I thought it might be a good idea to write a Flask API. This arrangement allows us access to Python’s ecosystem, and provides a (relatively) simple means by which we can consume the data in Excel using the Power Query Web Connector.

This blog is divided into two parts. In Part I, I describe the YFinanceService. This is a basic Flask service with APIs to perform the stock valuation calculation and more generally to obtain ticker data using yfinance.

In Part II, I describe how to use the YFinanceService from Excel Power Query to automate the process of performing a stock valuation calculation and to obtain a simple table of ticker data.

There are several alternative approaches. One could try a custom Python add-in for Excel. If you have non-financial data perhaps use Excel’s built-in Python support. The approach adopted here is a simple and quite generic way to have access to Python and its ecosystem. It allows you to obtain data and invoke / perform calculations.

Calculate the intrinsic value of a stock using Graham’s formula

In general, calculating stock values is quite involved. There are a number of different methods each with their advantages and disadvantages. For example Simply Wall St. describes a number of more complicated approaches. The approach adopted here is to use Graham’s formula to calculate the intrinsic value of a stock with a view to determining whether it is cheap or expensive compared to its current price. The Python code is as follows:

"""
Estimates the intrinsic value of a stock
"""


def stock_valuation_graham(eps: float, pe_base: float, g: float, avg_yield: float, cur_yield: float) -> float:
    """
    Calculate the intrinsic value of a stock using Graham's formula
    Reference: https://stablebread.com/how-to-calculate-the-intrinsic-value-of-a-company-like-benjamin-graham/

        V = (eps * (pe_base + 2g) * avg_yield) / cur_yield

    where:
    V = intrinsic value per share (over the next 7-10 years)
        EPS = earnings per share (over the trailing twelve months (TTM))
        8.5 = price-to-earnings (P/E) base for a no-growth company
        g = reasonably expected annual growth rate (over the next 7-10 years)
        avg_yield = average yield of AAA Corporate Bonds
        cur_yield = current yield of AAA Corporate Bonds
    """

    V = (eps * (pe_base + 1.1 * g) * avg_yield) / cur_yield

    return V

The formula itself is self-explanatory. A more detailed explanation of the calculation is available here. The parameters are divided into those values that we obtain per ticker and some global values. Thus, the EPS and the expected annual growth rate are obtained from the ticker info using yfinance. On the other hand, the average yield of AAA Corporate Bonds, and the current yield of AAA Corporate Bonds are setup as input parameters.

Finally, both the price-to-earnings (P/E) base for a no-growth company and the factor applied to the expected annual growth are hardcoded. The price-to-earnings (P/E) base for a no-growth company is established at 8.5. The factor applied to the expected annual growth - 2g seemed somewhat aggressive so we lower our expectations to 1.1g. Later, both these values may be set up as input parameters.

The YFinanceService

The YFinanceService consists of four main APIs (endpoints):

• TickerInfo
• TickerData
• IntrinsicValue
• IntrinsicValues

These are declared in YFinanceService.py.

# Flask application
app = Flask(__name__)
api = Api(app)

# Register Endpoints
api.namespace('YFinanceService', description='YFinance Service API')

api.add_resource(VersionInfo, '/YFinanceService/VersionInfo')
api.add_resource(TickerInfo, '/YFinanceService/TickerInfo')
api.add_resource(TickerData, '/YFinanceService/TickerData')
api.add_resource(IntrinsicValue, '/YFinanceService/IntrinsicValue')
api.add_resource(IntrinsicValues, '/YFinanceService/IntrinsicValues')


if __name__ == '__main__':
    DEBUG = False
    HOST = environ.get('SERVER_HOST', 'localhost')
    try:
        PORT = int(environ.get('SERVER_PORT', '5000'))
    except ValueError:
        PORT = 5000
    # run our Flask app
    app.run(HOST, PORT, DEBUG)

Running the YFinanceService

The YFinanceService can be run from the command line. Locate the \YFinanceService subdirectory. At the command prompt, type python YFinanceService.py. Alternatively, the service can be run directly with the debugger from within VSCode.

You should see a message similar to the following:

To test the service from a browser, type the following in the address bar:

http://localhost:5000/YFinanceService/VersionInfo

and you should see the json response packet: {"VersionInfo": {"flask": "3.1.0", "yfinance": "0.2.54", "pandas": "2.2.3"}}

YFinanceService Endpoints

With the service running, the API endpoints can be exercised via a browser using the examples below (assuming the service is running on port 5000).

IntrinsicValue API

The Flask API IntrinsicValue only supports the GET method. The other REST methods (PUT, POST, etc.) are not defined. The GET API takes three parameters:

• ticker - The ticker symbol
• avg_yield - The average yield of AAA Corporate Bonds
• cur_yield - The current yield of AAA Corporate Bonds

The GET function is divided into two parts. The first part obtains the (required) parameters. The second part calls a wrapper function stock_valuation.

class IntrinsicValue(Resource):
    """ Calculate the intrinsic value of a single stock using Graham's formula. """
    def get(self):

        parser = reqparse.RequestParser()

        parser.add_argument("ticker", type=str, required=True, help="The ticker symbol of the stock")
        parser.add_argument("avg_yield", type=float, required=True, help="Average yield of AAA Corporate Bonds")
        parser.add_argument("cur_yield", type=float, required=True, help="Current yield of AAA Corporate Bonds")

        args = parser.parse_args()

        symbol: str = args.get("ticker")
        avg_yield: float = args.get("avg_yield")
        cur_yield: float = args.get("cur_yield")

        v: float = stock_valuation(symbol, avg_yield, cur_yield)

        return {"value": v}, 200  # return info and 200 OK code

The stock_valuation function obtains the ticker.info dictionary from yfinance, and uses this to get the trailing EPS. The price-to-earnings (P/E) base for a no-growth company is hardcoded at 8.5. The expected growth rate g is obtained from the Yahoo finance stock info library. Finally, the stock_valuation_graham function is called with all the required parameters and the intrinsic value is returned.

For example, to determine the instrinsic value of GM we can use the following query: http://localhost:5000/YFinanceService/IntrinsicValue?ticker=GM&avg_yield=4.25&cur_yield=3.25.

This returns a dictionary with a single value: {“value”: 82.07549}. Comparing this to the current price of GM (46.72), we might conclude that the stock is undervalued.

The stock_valuation function is also used by the IntrinsicValues API. This does the same as the IntrinsicValue API but for a list of tickers. For this reason, the stock_valuation function handles any errors and exceptions and returns 0.0 for example if a ticker is not found or the data item is not available. If this had been part of a web application, we might have returned an error code and some exception message via the API into an HTML page. However, in this arrangement, the client is Excel, and specifically the Power Query Web Connector. So to simplify the client (Excel) processing we have tried to make the API calls as resilient as possible.

For example, the following: http://localhost:5000/YFinanceService/IntrinsicValues?tickers=SAN.MC,BBVA.MC,IBE.MC,AENA.MC&avg_yield=3.25&cur_yield=1.25 queries for the intrinsic values of four Spanish stocks. The results are returned as a comma-separated table with two columns: the ticker and the intrinsic value.

TickerData API

The Flask API TickerData also only supports the GET method. The purpose of the API is to retrieve one or more fields from the yfinance ticker info for a given list of ticker symbols. The output is a csv string that can be parsed into a table with the column headers being the field names and the rows being the data corresponding to each ticker.

class TickerData(Resource):
    """ Retrieve the specified fields from the input tickers """
    def get(self):

        parser = reqparse.RequestParser()

        parser.add_argument("tickers", type=str, action='split', required=True, help="One or more ticker symbols")
        parser.add_argument("fields", type=str, action='split', required=True, help="One or more fields")

        args = parser.parse_args()

        symbols: list = args.get("tickers")
        fields: list = args.get("fields")

        headers: list = ['Ticker']
        for field in fields:
            headers.append(field)

        rows: list = []

        try:

            for symbol in symbols:
                rows.append(get_items(symbol, fields))

        except Exception:
            print("An error occured")

        df = pd.DataFrame(rows, columns=headers)

        return df.to_csv(index=False, header=True), 200  # return info and 200 OK code

As previously described, the API is divided into two parts. The first part obtains the parameters, checks that the required parameters are present and sets up the table headers. The second part iterates over the input list of ticker symbols and for each symbol requests the data items from the ticker.info structure. Each row is appended to a list of rows. Finally, the csv table is returned using the DataFrame.to_csv method.

The parameters are a comma-separated list of ticker symbols and a comma-separated list of fields (key names in the ticker.info dictionary). This provides a relatively flexible way to obtain stock financial data.

For example to obtain the following indicators:

• trailingPegRatio
• trailingPE
• forwardPE

for the symbols: MSFT, AMZN, WAL and TSLA we can create the following query: http://localhost:5000/YFinanceService/TickerData?tickers=MSFT,AMZN,WAL,TSLA&fields=trailingPegRatio,trailingPE,forwardPE

As before, the error handling is resilient rather than informative. The table will be built with empty values if there is a problem obtaining the data, if the key doesn’t exist, or if the ticker symbol is unavailable and so on. This is not necessarily ideal but it simplifies the client processing.

Wrap Up

This blog has introduced the YFinanceService, a simple Flask service that defines some endpoints to allow us to perform a stock valuation calculation and more generally to obtain ticker data using the Python yfinance library. In Part II of the blog, I describe how to use the raw APIs in Excel via the Power Query Web Connector.

The first part of this two-part blog described the YFinanceService. This is a Flask service that wraps calls to the Python yfinance library and which served up some of the data via the APIs (/IntrinsicValue, /TickerData and so on). Part II of the blog describes how to make use of the APIs in an Excel spreadsheet, specifically using the Power Query Web Connector to retrieve the data.

Consuming the APIs from Excel

One of the main objectives of this project was to be able to obtain the data from yfinance via the YFinanceService and use it in Excel. For this, we use the Stock Valuation Spreadsheet.

The spreadsheet uses the Power Query Web Connector. However, rather than using the Web Connector directly to call our YFinanceService APIs, we use Power Query to create blank queries and write the Power Query M language code to obtain the parameters from the worksheets and to process the results into a table.

The spreadsheet consists of the following worksheets:

• Setup - defines a table (Parameters) with global model parameters (average yield and current yield). This table is passed into the Power Query queries when needed.
• Screener - this sheet obtains the current prices for a number of stocks and calculates the intrinsic value for each, displaying a recommendation alongside the computed safety margin. The margin of safety is discussed in the article How to Calculate the Intrinsic Value of a Company Like Benjamin Graham.
• IntrinsicValues - this sheet defines a list of tickers and requests the intrinsic values for all of them. Similar to the above, it displays a recommendation alongside the computed safety margin.
• Ticker Data - this sheet requests a number of key indicators for a list of stock tickers and displays the results.

After opening the spreadsheet, select the Setup worksheet and from the Data menu select Queries and Connections. This will list the queries that are used in this sheet. The three main queries are:

• KeyIndicators (in the TickerData worksheet)
• IntrinsicValues (in the IntrinsicValues worksheet)
• CalcIntrinsicValue which depends on the function fnCalcIntrinsicValue (in the Screener worksheet)

KeyIndicators Query

The TickerData worksheet sets up two tables: TableSymbols and the TableFields. These are used as the inputs to the KeyIndicators query. The Power Query M code builds up the API query string from these inputs and processes the result to produce a table. The query can be refreshed using the refresh symbol on the right-hand side of the Queries & Connections panel.

IntrinsicValues Query

The IntrinsicValues worksheet sets up a table called TickerList. Additionally the query obtains the values for the average yield (avg_yield) and the current yield (cur_yield) from the parameters table defined in the Setup worksheet. With these values it constructs the API query string and calls the appropriate YFinanceService. If the query succeeds, the resulting string is processed into a table with two columns: Ticker and Intrinsic Value.

CalcIntrinsicValue Query

The CalcIntrinsicValue query depends on the fnCalcIntrinsicValue query/function. This in turn depends on a query (IntrinsicValue) which has had a parameter (Ticker) added to it. The parameter is added using the Manage Parameters menu on the Home menu strip. Once the parameter is added we can convert the original query to a function - fnCalcIntrinsicValue. The function can be invoked either by double clicking it or by entering a parameter, the ticker symbol in this case, and pressing “Invoke”. However, we will not use this method of calling the function.

Instead, we create a further query CalcIntrinsicValue. This creates a new table. The table uses the tickers defined in the TableTickers (input) table and adds a new custom column. The custom column invokes the intrinsic value function for each ticker value.

	// Add custom IntrinsicValue column based on the function fnCalcIntrinsicValue parameterised with each of the Tickers
    InvokedCustomFunction = Table.AddColumn(TickerSource, "IntrinsicValue", each fnCalcIntrinsicValue([Tickers])),

The function fnCalcIntrinsicValue returns a key-value pair { value: nnnn }, so we expand the result and use only the second component:

    ExpandedIntrinsicValue = Table.ExpandRecordColumn(InvokedCustomFunction, "IntrinsicValue", {"value"}, {"IntrinsicValue.value"}),

The resulting table is processed to display on the Intrinsic Value column (which is what appears in the worksheet).

A word about performance

Overall the performance of the queries is slow. Assuming the YFinanceService is already running, and the Excel spreadsheet is open. Then, pressing the refresh button on any of the queries needs to 1) construct a query string, 2) call the API via Web.Contents, and 3) process the results. The call via Web.Contents calls a Flask API which wraps Python code, and in this case calls via yfinance and yahoo_fin. Both these use web scraping and this is what determines the slow performance.

Wrap Up

In Part I we described the YFinanceService and the API calls to the Python yfinance library. Part II described how to make use of the APIs in an Excel spreadsheet, specifically using the Power Query Web Connector to retrieve the data.

Overall the arrangement that we have is:

• The YFinanceService API query (“What is the intrinsic value of this stock using Graham’s formula?”) takes its parameters and inputs from an Excel table and invokes the stock valuation function via the Power Query Web Connector.
• The calculation is done in Python using data from the yfinance library.
• The result is served up over Flask endpoints (the API).
• The result data is massaged using the Power Query Web Connector and used in the Excel worksheet for additional calculations.

Advantages and Disadvantages

There are advantages and disadvantages to this approach. The main advantages are:

• it is simple to debug (on the Python side); somewhat less so using the Power Query M language.
• it is relatively simple to extend: you can add more functionality on the Python side, and/or extend the Flask API to support additional endpoints that expose more of the data from yfinance. The disadvantages are mainly concerned with the performance.
• the performance is somewhat disappointing. Specifically, in this case we needed some data from another finance package yahoofinance. The underlying mechanism requires web-scraping specific yahoo finance pages (and tables and elements etc.) which results in relatively slow performance.

Recently, I’ve been using Python’s yfinance library to perform some simple portfolio analysis. Rather than using a Jupyter notebook to run the Python code and visualise the results, I thought it might be a nice idea to build a .NET desktop application using WinUI 3.0, XAML and C#. The application allows you to define a portfolio and manages the presentation of the results graphically. The underlying analysis is performed using Python. In order to call the Python code from C#, I use Python.NET. The purpose of this blog post is to describe how to use Python.NET to call Python functions from a C# application. The complete PortfolioManager application is available on github.

The main advantage of this approach is that it allows access to Python’s extensive ecosystem from .NET. In this specific case, I wanted to build a simple portfolio management application while at the same time taking advantage of Python’s yfinance library to get the financial data I wanted and to use pandas and numpy to perform the calculations. I also wanted a slightly more real-world project using Python.NET. Information about this is somewhat limited and the examples are a little sketchy.

Versions

We use the following versions:

• Python 3.12
• yfinance 0.2.54
• Python.NET 3.0.5

The Python API

To start off, I wrote a simple portfolio analysis script. The script is based on a portfolio management app written in Python. However, I refactored the main functionality in order to separate the calculations and analysis from the (graphical) presentation layer. This allowed me to call the Python functions from C# to obtain the data and then process the (converted) data in the C# application.

The API consists of four functions.

1. portfolio_returns(ticker_values, start_date, end_date) This function takes the user-supplied tickers (and values), start date and end date. The function computes the ticker weights and obtains the adjusted close prices (if available) for each ticker using the yfinance library. The function returns a dictionary with the ticker weights, a DataFrame containing the individual ticker returns, and the overall portfolio returns.

2. perform_portfolio_analysis(data, ticker_weights, risk_free_rate) This function takes the portfolio returns DataFrame computed previously, the ticker weights, and a user-supplied risk free rate and uses this to calculate individual security returns, cumulative returns, volatility, and Sharpe Ratios. These are returned in a dictionary.

3. benchmark_returns(benchmark, start_date, end_date) This function obtains the benchmark data from yfinance using the supplied parameters. It also returns a dictionary.

4. portfolio_vs_benchmark(port_returns, benchmark_returns, risk_free_rate) This function calculates the cumulative returns, annualized volatility, and Sharpe Ratios for both the portfolio and the benchmark using the data obtained in the previous functions. It provides a side-by-side comparison to assess the portfolio’s performance relative to the benchmark.

These four functions define the API that is used by the .NET application.

The .NET Application

Prerequisites

For the .NET application we first need to download and build Python.NET. The code is available from https://github.com/pythonnet/pythonnet. Once downloaded, open the solution file (pythonnet.sln), and select the Python.Runtime as the startup project. Then right-click and build the debug version. If the build is successful the Python.Runtime.dll will be located in the pythonnet directory under \pythonnet\runtime.

The PortfolioManager Application

Having built the Python.Runtime.dll, we need to add it to the Portfolio Manager application. This is a WinUI 3.0 (C#/XAML) application which makes use of the Windows Community Toolkit for dependency injection and the MVVM architecture. It also uses an Sqlite database to manage a collection of one or more portfolios.

Aside from the test framework (PortfolioManager.Tests.MSTest), the solution consists of two main projects: the PortfolioManager and PortfolioManager.Core. The PortfolioManager project deals with the application logic, user settings, the views and the view models. The reusable logic and data models are located in the PortfolioManager.Core project. So we add the Python.Runtime.dll to the Dependencies here under the Assemblies node.

The Python service

In the core library, we define a simple IPythonService interface that encapsulates the initialisation and the calls to the Python API.

public interface IPythonService
{
    void Initialize(string pythonDll);

    bool RunPortfolioAnalysis(PortfolioItem portfolio);
}

In the PortfolioManager project, in the App.xaml.cs, we add the IPythonService to the list of services to be activated

...
services.AddSingleton<IPythonService, PythonService>();

Finally, we pass an instance of the IPythonService into the PortfolioDetailViewModel. This view model manages the graphs output as a result of running the portfolio analysis. When the view is activated after clicking on the ‘Run’ button, the PythonService can be initialised and the view model can run the portfolio analysis based on the selected portfolio.

    public PortfolioDetailViewModel(IPythonService pythonService)
    {
        _pythonService = pythonService;

        var settings = App.GetService<SettingsViewModel>();

        var pythonDll = settings.PythonLibrary;
        _pythonService.Initialize(pythonDll);

		//...
    }

Initialisation of the PythonService consists of setting the path to the Python dll. This is configured via the application settings page. Additionally, we set up a couple of empty output streams to redirect any console output from the Python.Runtime.

Portfolio Analysis

The main function RunPortfolioAnalysis is called with the currently selected PortfolioItem. The PortfolioItem contains the user supplied parameters which have been obtained from the UI.

The RunPortfolioAnalysis function loads the portfolio_analysis.py script and passes it to Python.NET, which creates a Python module object. The module object is used to access the script functions and variables. We use C#’s ‘dynamic’ keyword to delay the binding. We can see the available dynamically bound functions and parameters if we expand the module variable in the Watch window.

The portfolio analysis proceeds by calling the Python functions using the user-supplied parameters and any intermediate results. Once these have been appropriately converted they are copied into the portfolio item’s data members.

	...
	dynamic portfolioResults = module.portfolio_returns(tickers, startDate, endDate);

	var pyTickerWeights = new PyDict(portfolioResults["weights"]);

	portfolio.TickerWeights = Interop.Converters.ConvertTickerWeights(pyTickerWeights);
	...

In this snippet, the Python object pyTickerWeights is extracted from the dictionary returned from calling module.portfolio_returns. The Python ‘dict’ is converted into a C# List<TickerWeight>. The ConvertTickerWeights function just iterates over each ticker and obtains the corresponding weight. This is then assigned to the TickerWeights member of the portfolio item. The TickerWeights list is then used in a Pie Chart that displays the percentage allocation of tickers in the portfolio.

Once the analysis is complete, the PortfolioDetailViewModel initialises the PlotModel with the data from the portfolio item, which then updates the graph view as it is bound to the specific PlotModel.

It is worth highlighting that the performance is quite slow. There seem to be two main aspects to this:

1. The initialisation of the Python engine seems to take some time, and perhaps this could be done once at startup.
2. The PortfolioDetailViewModel renders six graphs with different datasets serially. It seems to me that this might be better refactored to use C# asynchronous processing.

I am currently investigating both these issues.

The interop layer

This layer provides facilities to convert between C# and Python types. Basic types (strings, ints, doubles etc.) are converted transparently. From C#, we use the .ToPython() extension method to convert to a Python object. And from Python, we use the .AsManagedObject(typeof(T)) function to return a C# object which we can then cast. Converting from a pandas DataFrame or series can be somewhat more involved.

Wrap-up

This blog post has described how to call Python functions from a C# desktop application using Python.NET. This arrangement (WinUI 3.0 C# front-end, Python.NET and Python) has the advantage of flexibility. The approach is quite generalisable. I could just as easily have created a TensorFlow model and called it from a C# application, if that had been my interest. On the other hand, the main disadvantage is the slow performance. This is an area that I am still currently investigating.

If you are an R/RStudio user and you are learning C/C++ then this blog post may be interesting. Suppose that you’ve written a function in C/C++ (or even a whole program). You know how to run your code from the command line, passing in arguments and so on. However, this is not necessarily very convenient. Wouldn’t it be nice to have your C/C++ functionality available in R? Well you can. In a few easy steps (using Rcpp) you can make your C/C++ functions available in R/RStudio.

This blog post describes the steps required. I won’t dwell too much on the technicalities. The details are covered elsewhere: Rcpp. What I’m interested in here is setting up an Rcpp package that can call C++ functions from R/RStudio.

This project is no more than a starting point. Later on, I may add bells-and-whistles. The complete FinancialFunctions project is available on GitHub FinancialFunctions.

We’ll start off by creating an Rcpp package. This will give us a framework that we can use to drop in the C++ code. Then, we will tweak the Rcpp code to call our C++ functions and build the package. For reference we are using the following versions:

• R version 4.4.1
• RStudio version 2024.04.2 (Build 764)
• RTools version 4.4

1. Creating an Rcpp package

Follow these steps to create an Rcpp package.

First, make sure Rcpp is installed. If not, just run the following command in RStudio: install.packages("Rcpp")

• Select File > New Project …
• In the New Project Wizard dialog, select New Directory
• In the Project Type page, scroll down the list of project types and select ‘R Package using Rcpp’.

• In the ‘Create R Package using Rcpp’ page, fill in the Directory name. In this case it is FinancialFunctions.
• Press the Create Project button. A new RStudio session is started, and the project opens with a helpful Read-and-delete-me file.

Navigate to the \src directory and open the file rcpp_hello_world.cpp. Here there is a test C++ function called rcpp_hello_world() which we can call from R after we build the package.

• Select the menu: Build > Build binary package

If everything goes okay, we should see a message similar to this:

...
packaged installation of 'FinancialFunctions' as FinancialFunctions_1.0.zip
─  DONE (FinancialFunctions)
[1] "D:/TEMP/FinancialFunctions_1.0.zip"

Binary package written to D:/TEMP

Now we have built a fully functional Rcpp package. Open the zip-file and copy the contents into your R library directory (e.g. D:\R\R-4.4.1\library). Then, in RStudio, run the following script:

> library(FinancialFunctions)		# Load the package
> results <- rcpp_hello_world()		# Call the C++ function
> results							# Print the results
[[1]]
[1] "foo" "bar"

[[2]]
[1] 0 1

2. Adding C++ code

Now that we have a working skeleton, we can add our own C++ functions to the package.

Copy the two files FinancialFunctions.h and FinancialFunctions.cpp into the \src directory.

There are two functions declared in the header file: price and ytm (yield-to-maturity).

The price function computes the present value of a stream of periodic cashflows using a single discount rate. The yield-to-maturity function computes the single rate of return implied by the periodic cashflows and the price (present value).

The price function takes a vector of times and a vector of cashflows (for a bond these are the coupons and the final principal repayment). In addition, the function takes a single rate, which is used to discount all the cashflows.

The yield-to-maturity function also takes a vector of times and a vector of cashflows (as above) and a (bond) price. From this, it iteratively determines the single rate at which the periodic stream of cashflows should be discounted to obtain the supplied price.

3. Connecting R with C++ using Rcpp

Delete the file rcpp_hello_world.cpp as we no longer need the test function. Then copy the file rcpp_functions.cpp into the \src directory. This file contains the functions that we want to export from C++ to R using Rcpp.

At the top of the file, we include the main Rcpp header file. This is followed by the STL vector include and finally, we include the declarations of our functions in FinancialFunctions.h.

The code in rcpp_functions.cpp declares an Rcpp function called bond_price. Most importantly, it is preceded by the line // [[Rcpp::export]]. This attribute indicates that the function should be exported, that is, made visible to R. As simple as that. Rcpp takes care of all the details. All we are concerned about is what parameters we are passed and how to call our function and get the results.

The bond_price function takes as its parameters two numeric vectors (the times and the cashflows), and a rate (a double). These are passed in from R. The function returns a double value, the calculated price. The Rcpp function looks like this.

// [[Rcpp::export]]
double bond_price(Rcpp::NumericVector times, Rcpp::NumericVector cashflows, double rate)
{
  std::vector<double> _times = Rcpp::as<std::vector<double>>(times);
  std::vector<double> _cashflows = Rcpp::as<std::vector<double>>(cashflows);
  
  double price = FinancialFunctions::Bonds::price(_times, _cashflows, rate);
 
  return price;
}

Inside the function, the Rcpp types NumericVector are converted to std::vector<double>, which is what our function expects.

Finally we call our function and return the result.

The Rcpp code for bond_ytm is very similar.

Everything is now ready to be built.

• Select the menu: Build > Build binary package.

Keep our fingers crossed that we have not made any mistakes. If everything goes okay, we should see a message similar to this:

Binary package written to D:/Development/Projects/R

Copy the contents of the zip file into your R library directory as previously, overwriting the initial version of the package.

In order to check that everything works as expected, run the following script.

library(FinancialFunctions)
# Example: a 3 year bond with a face value of $100 makes annual coupon 
# payments of 10%. The current interest rate (with annual compounding) is 9%.
times <- c(1, 2, 3)
cashflows <- c(10, 10, 110)
rate <- 0.09

# Calculate price and yield to maturity
price <- FinancialFunctions::bond_price(times, cashflows, rate)
ytm <- FinancialFunctions::bond_ytm(times, cashflows, price)

# Display the results
price
ytm

There should be no errors. We can check the results against the manual calculations in a spreadsheet if required.

Wrap up.

That’s all there is to it. We now have a working Rcpp package that calls C/C++ functions from R.

But that really is only the beginning. There’s a lot more that can be done. For starters, we could improve the project design by isolating our code in a static library. We could also add some test cases, perhaps using a package like ‘testthat’. Finally, we could add a couple more functions: perhaps bond duration and convexity calculations. I might look at how to do all this in a future article. The complete project is available on GitHub, FinancialFunctions.