Appendix B. Software

Table A.1: A list of indicative free or open-source packages, libraries, and toolboxes linking to the theory sections of this article. The authors assume no liability for the software listed below; interested users are strongly advised to read the respective documentations and licences terms.

Related section Software Package/Library/Toolbox Function(s) Comments
§2.2.1. Box-Cox transformations R forecast BoxCox; InvBoxCox; BoxCox.lambda; Functions to transform the input variable using a Box-Cox transformation, reverse the transformation and find optimal parameters. https://cran.r-project.org/package=forecast
§2.2.2. Box-Cox transformations Time series decomposition R stats decompose; stl Classical decomposition method (additive and multiplicative), and STL decomposition method. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html
R forecast seasadj; seasonal; mstl; msts; tbats.components Tools for extracting components, and multiple seasonal decomposition methods. https://cran.r-project.org/package=forecast
R tsutils decomp; seasplot Classical decomposition method, and functions for seasonal plots. https://cran.r-project.org/package=tsutils
R stR AutoSTR; STR; heuristicSTR; plot.STR; seasadj.STR Seasonal-Trend decomposition based on Regression. https://cran.r-project.org/package=stR
R seasonal seas Functions for X-11, SEATS, and X-13-ARIMA-SEATS decomposition methods. https://cran.r-project.org/package=seasonal
Gretl buys_ballot Plot seasonal time series components. http://gretl.sourceforge.net/
Gretl season_plot set_season_plot Plot seasonal time-series components. http://gretl.sourceforge.net/
Gretl tsfcst decompcl Classical time series decomposition. http://gretl.sourceforge.net/
Gretl StrucTiSM STSM_components Decomposition using structural timeseries model. http://gretl.sourceforge.net/
§2.2.3. Anomaly detection and time series forecasting R anomalize time_decompose; anomalize; time_recompose A “tidy” workflow for detecting anomalies in data. https://cran.r-project.org/package=anomalize
R oddstream find_odd_streams; extract_tsfeatures; set_outlier_threshold Early detection of anomalous series within a large collection of streaming time series data. https://cran.r-project.org/package=oddstream
R tsoutliers tso; locate.outliers.oloop; remove.outliers Detection of outliers in time series such as Innovational outliers, additive outliers, level shifts, temporary changes and seasonal level shifts. https://cran.r-project.org/package=tsoutliers
R stray find_HDoutliers; find_threshold; display_HDoutliers Anomaly detection in high dimensional and temporal data. https://cran.r-project.org/package=stray
R forecast tsoutliers; tsclean Provides some simple heuristic methods for identifying and correcting outliers. https://cran.r-project.org/package=seasonal
R OutliersO3 OutliersO3; O3plotM; O3plotT; O3prep Draws overview of outliers (O3) Plots. https://cran.r-project.org/package=OutliersO3
R CRAN Task View Anomaly Detection with R Contains a list of R packages that can be used for anomaly detection. https://github.com/pridiltal/ctv-AnomalyDetection
Gretl tramolin Outlier detection/correction and missing data interpolation. http://gretl.sourceforge.net/
§2.2.4. Robust handling of outliers in time series forecasting R gets isat Function for running impulse and step indicator saturation. https://cran.r-project.org/package=gets
§2.3.1. Exponential smoothing models R forecast ets; forecast.ets; ses; Functions for simple exponential smoothing and automatic exponential smoothing modelling. https://cran.r-project.org/package=forecast
R smooth es Function for automatic exponential smoothing modelling. https://cran.r-project.org/package=smooth
Gretl tsfcst expsmpars Simple exponential smoothing minimising the sum of squared errors. http://gretl.sourceforge.net/
§2.3.2. Time-series regression models R stats lm Fitting linear regression models. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/00Index.html
R leaps regsubsets Functions for selecting linear regression models. https://cran.r-project.org/package=leaps
R relaimpo Relative importance of regressors in linear models. https://cran.r-project.org/package=relaimpo
R MASS stepAIC Choose a model by AIC in a stepwise algorithm. https://cran.r-project.org/package=MASS
Gretl ols, lad, midasreg Regression models with OLS, LAD, and MIDAS with functionality for forecasting. http://gretl.sourceforge.net/
§2.3.3. Theta method and models R forecast thetaf Returns forecasts and prediction intervals for a theta method forecast. https://cran.r-project.org/package=forecast
R forecTheta stheta; stm; otm; dstm; dotm Functions for forecasting univariate time series using Theta Models. https://cran.r-project.org/package=forecTheta
R tsutils theta Estimate Theta method. https://cran.r-project.org/package=tsutils
Gretl tsfcst stheta Theta-method for univariate forecasting. http://gretl.sourceforge.net/
§2.3.4. Autoregressive integrated moving average (ARIMA) models R forecast auto.arima; Arima; arfima; arima.errors; arimaorder Functions for fitting and forecasting with ARIMA models. https://cran.r-project.org/package=forecast
R smooth auto.msarima; auto.ssarima; msarima; ssarima State-space and multiple seasonalities implementations of ARIMA models. https://cran.r-project.org/package=smooth
Gretl arima Functions for fitting and forecasting with SARIMAX models. http://gretl.sourceforge.net/
Gretl auto_arima Find best fitting SARIMAX model with functions for forecasting. http://gretl.sourceforge.net/
Gretl armax Automatically determine the best ARMAX model. http://gretl.sourceforge.net/
R aTSA adf.test Augmented Dickey-Fuller test. https://cran.r-project.org/package=aTSA
R tseries kpss.test Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test. https://cran.r-project.org/package=tseries
R forecast ndiffs; nsdiffs Estimates the number of (seasonal) differences required to make a given time series stationary. https://cran.r-project.org/package=forecast
Gretl adf; adf-gls; kpss; levinlin Various unit-root tests for time-series and panel data. http://gretl.sourceforge.net/
§2.3.4. Autoregressive integrated moving average (ARIMA) models (continued) Gretl DHF_test Package for Dickey-Hasza-Fuller seasonal Unit Root Test. http://gretl.sourceforge.net/
Gretl DP Package for testing for a double unit root. http://gretl.sourceforge.net/
Gretl GHegy Package Seasonal unit roots tests. http://gretl.sourceforge.net/
Gretl Kapetanios Package for Kapetanios’ unit root test with possible structural breaks. http://gretl.sourceforge.net/
Gretl PPtest Package for running Phillips-Perron unit root test. http://gretl.sourceforge.net/
Gretl VSG_test Package for test proposed by Ventosa-Santaulària and Gómez-Zaldívar. http://gretl.sourceforge.net/
§2.3.5. Forecasting for multiple seasonal cycles R smooth msarima; ssarima Functions for forecasting data with multiple seasonal cycles. https://cran.r-project.org/package=smooth
R fable model; forecast; fasster; ETS; ARIMA; TSLM Forecasting models for tidy time series. https://cran.r-project.org/package=fable
R, Python prophet Facebook’s automatic forecasting procedure. https://cran.r-project.org/package=prophet
R tidymodels Collection of packages for modelling and machine learning using tidyverse principles. https://cran.r-project.org/package=tidymodels
R forecast tbats; dshw Functions for forecasting data with multiple seasonal cycles. https://cran.r-project.org/package=forecast
R fable.prophet prophet; forecast A tidy R interface to the prophet forecasting procedure using fable. https://github.com/mitchelloharawild/fable.prophet
§2.3.6. State-space models Matlab SSpace General modelling of linear, non-linear and non-Gaussian State Space systems. https://github.com/djpedregal/SSpace
Matlab SSM General modelling of linear, non-linear and non-Gaussian State Space systems. https://www.mathworks.com/help/econ/ssm-class.html
Matlab SSMMATLAB A Set of MATLAB Programs for the Statistical Analysis of State Space Models. https://github.com/vgomezenriquez/ssmmatlab
Matlab E4 A MATLAB toolbox for time series analysis in State Space form. https://www.ucm.es/e-4
R UComp Automatic identification of Unobserved Components models in State Space form. https://cran.r-project.org/package=UComp
R statespacer State Space modelling, mainly ARIMA and Basic Structural Models. https://cran.r-project.org/package=statespacer
R smooth Forecasting using single error State Space models. https://cran.r-project.org/package=smooth
R bssm Bayesian Inference of Non-Gaussian State Space Models. https://cran.r-project.org/package=bssm
R mssm Multivariate State Space models. https://cran.r-project.org/package=mssm
R KFAS Kalman Filter and Smoother for Exponential Family State Space Models. https://cran.r-project.org/package=KFAS
§2.3.6. State-space models (continued) R TSSS Time Series Analysis with State Space Model, based on the methods in Kitagawa (1993). https://cran.r-project.org/package=TSSS
R dlm Bayesian and Likelihood Analysis of Dynamic Linear Models (Gaussian State Space models). https://cran.r-project.org/package=dlm
Python statsmodels statespace Time Series Analysis by State Space Methods. https://www.statsmodels.org/stable/index.html
Gretl kfilter; ksmooth; kdsmooth; ksimul State Space Modeling functionality with function for forecasting. http://gretl.sourceforge.net/
Gretl StrucTiSM STSM_fcast Harvey-style Structural Time Series Models with function for forecasting. http://gretl.sourceforge.net/
§2.3.7. Models for population processes R dembase General-purpose tools for demographic analysis. https://github.com/StatisticsNZ/dembase
R demest Bayesian statistical methods for demography. https://github.com/StatisticsNZ/demest
R demlife Creating and working with life tables. https://github.com/StatisticsNZ/demlife
R BayesPop Generating population projections for all countries of the world using several probabilistic components, such as total fertility rate and life expectancy. https://cran.r-project.org/package=bayesPop
R bayesTFR Making probabilistic projections of total fertility rate for all countries of the world, using a Bayesian hierarchical model. https://cran.r-project.org/package=bayesTFR
R bayesLife Making probabilistic projections of life expectancy for all countries of the world, using a Bayesian hierarchical model. https://cran.r-project.org/package=bayesLife
Spreadsheet DAPPS Demographic Analysis and Population Projection System: Standalone spreadsheet-based software for demographic estimation and projections, prepared by the US Census Bureau. https://www.census.gov/data/software/dapps.Overview.html
§2.3.9. Forecasting with many variables R gets getsm, getsv, isat, isatvar Package that implements general to specific model selection, indicator saturation, with functionality for forecasting. https://cran.r-project.org/package=gets
R vars Functions and routines for VAR Modelling. https://cran.r-project.org/package=vars
Gretl var; system Fitting system-models with functionality for forecasting. http://gretl.sourceforge.net/
§2.3.10. Functional time series models R ftsa ftsm; farforecast; T_stationarity Functional time series analysis. https://cran.r-project.org/package=ftsa
§2.3.11. ARCH/GARCH models R tseries garch Fit GARCH models to time series. https://cran.r-project.org/package=tseries
Python PyFlux Time series analysis and prediction tools that focus on autoregressive methods (ARIMA, ARCH, GARCH, etc). https://pyflux.readthedocs.io/en/latest/index.html
Gretl arch, garch Fit (G)ARCH models to time series. http://gretl.sourceforge.net/
§2.3.11. ARCH/GARCH models (continued) Gretl gig gig_estimate; gig_var_fcast Estimate various types of GARCH models. http://gretl.sourceforge.net/
§2.3.12. Markov switching models R MSwM Fitting Markov switching models. https://cran.r-project.org/package=MSwM
R NHMSAR Non-homogeneous Markov switching autoregressive models. https://cran.r-project.org/package=NHMSAR
§2.3.13. Threshold models R TAR Bayesian modelling of autoregressive threshold time series models. https://cran.r-project.org/package=TAR
R TSA tar; star Functions for threshold models (and general time series analysis). https://cran.r-project.org/package=TSA
Gretl Threshold_Panel THRESH_SETUP Hansen’s panel threshold model. http://gretl.sourceforge.net/
Gretl SETAR Estimation of a SETAR model. http://gretl.sourceforge.net/
§2.3.15. Forecasting with DSGE models R BMR forecast Bayesian Macroeconometrics in R (BMR) is a package for estimating and forecasting Bayesian VAR and DSGE. https://github.com/kthohr/BMR
Matlab/GNU Octave Dynare Software platform for solving, estimating, and making forecasts with DSGE. https://www.dynare.org/
§2.3.18. Innovation diffusion models R DIMORA Estimation of Bass Model, Generalised Bass Model, GGM, UCRCD. https://cran.r-project.org/package=DIMORA
R diffusion diffusion Various diffusion models to forecast new product growth. Currently the package contains Bass, Gompertz and Gamma/Shifted Gompertz curves. https://cran.r-project.org/package=diffusion
§2.3.19. The natural law of growth in competition R LS2Wstat scurve An S curve function between two constant values. https://cran.r-project.org/package=LS2Wstat
§2.3.21. Estimation and representation of uncertainty R hdrcde cde Conditional kernel density estimation to produce marginal distributions (uncertainty forecasts). https://cran.r-project.org/package=hdrcde
R gamlss gamlss Semi-parametric models for uncertainty forecasting. https://cran.r-project.org/package=gamlss
R gamboostLSS mboostLSS Semi-parametric component-wise gradient boosting models for uncertainty forecasting. https://cran.r-project.org/package=gamboostLSS
Python scikit-learn GradientBoostingRegressor; RandomForestQuantileRegressor Machine learning models (gradient boosting trees and random forests) for quantile forecasting. https://scikit-learn.org/stable/
R quantreg rq; lprq; nlqr Estimation and inference methods for models of conditional quantiles. https://cran.r-project.org/package=quantreg
R EnvStats FcnsByCatPredInts; pointwise Functions for computing prediction intervals and simultaneous prediction intervals. https://cran.r-project.org/package=EnvStats
R rmgarch dccfit-methods; dccforecast-methods Multivariate GARCH Models (e.g., forecasting covariance matrix). https://cran.r-project.org/package=rmgarch
§2.3.22. Forecasting under fat tails R FatTailsR Functions for Kiener distributions and fat tails. https://cran.r-project.org/package=FatTailsR
§2.4.3. Bayesian forecasting with copulas R VineCopula Statistical analysis of vine copula models. https://cran.r-project.org/package=VineCopula
R cdcopula Covariate-dependent copula models. https://github.com/feng-li/cdcopula
§2.4.3. Bayesian forecasting with copulas (continued) R FactorCopula Factor Copula Models for Mixed Continuous and Discrete Data. https://cran.r-project.org/package=FactorCopula
§2.5.1. Leading indicators and Granger causality R lmtest grangertest Test for Granger causality. https://cran.r-project.org/package=lmtest
Gretl var; omit Standard Granger-causality test. http://gretl.sourceforge.net/
Gretl BreitungCandelonTest Breitung-Candelon test of frequency-wise Granger (non-)causality. http://gretl.sourceforge.net/
§2.5.3. Variable Selection R glmnet Generalised linear models with Lasso or elastic net regularisation. https://cran.r-project.org/package=glmnet
Gretl omit Sequential removing of variables to a model. http://gretl.sourceforge.net/
Gretl addlist Sequential addition of variables to a model. http://gretl.sourceforge.net/
Gretl regls Add-on for regularised least squares such as Ridge, Lasso and Elastic-Net. http://gretl.sourceforge.net/
Gretl fsboost fsreg Forward-stagewise boosted regression estimates with functionality for forecasting. http://gretl.sourceforge.net/
§2.5.4. Model Selection R gets Functions for automatic general to specific model selection. https://cran.r-project.org/package=gets
§2.5.5. Cross-validation for time-series data R forecast CVar k-fold Cross-Validation applied to an autoregressive model. https://cran.r-project.org/package=forecast
Gretl fcast Forecasting command with functionality for recursive-window forecasts. http://gretl.sourceforge.net/
§2.6.1. Forecast combination: a brief review of statistical approaches R forecastHybrid Functions for ensemble time series forecasts. https://cran.r-project.org/package=forecastHybrid
§2.7.2. Forecasting on distributed systems Database InfluxDB Scalable datastore for metrics, events, and real-time analytics. https://www.influxdata.com/time-series-database/
Database OpenTSDB A scalable, distributed Time Series Database. http://opentsdb.net/
Database RRDtool A program for easily maintaining a database of time-series data. https://oss.oetiker.ch/rrdtool/
Database Timely A time series database application that provides secure access to time series data. https://code.nsa.gov/timely/
Python, R & Spark darima Implementations of distributed ARIMA models on Spark platform. https://github.com/xqnwang/darima
§2.7.3. Agent-based models R SpaDES Spatially explicit discrete event simulation models. https://cran.r-project.org/package=SpaDES
§2.7.4. Feature-based time series forecasting R tsfeatures tsfeatures Methods for extracting various features from time series data. https://cran.r-project.org/package=tsfeatures
Python tsfresh Calculates a large number of time series characteristics, the so called features. Further the package contains methods to evaluate the explaining power and importance of such characteristics for regression or classification tasks. https://tsfresh.readthedocs.io/en/latest/
§2.7.4. Feature-based time series forecasting (continued) Matlab hctsa Code framework that enables the extraction of thousands of time-series features from a time series (or a time-series dataset). It also provides a range of tools for visualising and analysing the resulting time-series feature matrix. https://github.com/benfulcher/hctsa
Python pyopy Python binding for hctsa. https://github.com/strawlab/pyopy
R fforma Tools for forecasting using a model combination approach. It can be used for model averaging or model selection. It works by training a ‘classifier’ that learns to select/combine different forecast models. https://github.com/pmontman/fforma
R gratis Efficient algorithms for generating time series with diverse and controllable characteristics, which can be used as the training data in feature-based time series forecasting. https://cran.r-project.org/package=gratis
R seer Implementations of a novel framework for forecast model selection using time series features; FFORMS (Feature-based FORecast Model Selection). https://cran.r-project.org/package=seer
§2.7.5. Forecasting with bootstrap R bootstrap Various bootstrapping functions. https://cran.r-project.org/package=bootstrap
Gretl uniFCextensions uniFCboot Estimate an interval forecast without assuming Gaussian innovations. http://gretl.sourceforge.net/
§2.7.6. Bagging for time series forecasting R forecast baggedETS Returns forecasts and other information for bagged ETS models. https://cran.r-project.org/package=forecast
R tshacks baggedClusterETS; treatedETS Returns forecasts for bagged Cluster and Treated ETS models. https://github.com/tiagomendesdantas/tshacks
§2.7.8. Neural Networks Python, MXNet & PyTorch GluonTS Framework for building deep learning based models including a number of pre-built models such as feed-forward neural networks. https://github.com/awslabs/gluon-ts
R nnfor mlp; elm Time Series Forecasting with Neural Networks. https://cran.r-project.org/package=nnfor
Python neural prophet Reimplementation of prophet in PyTorch, and extensions to it. https://github.com/ourownstory/neural_prophet
Python RNNs for forecasting in Tensorflow. https://github.com/HansikaPH/time-series-forecasting
R ANN2 neuralnetwork Artificial Neural Networks. https://cran.r-project.org/package=ANN2
R nnet nnet Feed-Forward Neural Networks and Multinomial Log-Linear Models. https://cran.r-project.org/package=nnet
R forecast nnetar Auto-regressive Neural Network for time series forecasting. https://cran.r-project.org/package=forecast
§2.7.9. Deep Probabilistic models Python, MXNet & PyTorch GluonTS Framework for building deep learning based models including a number of pre-built models such as DeepAR, DeepState models and NBEATS. https://github.com/awslabs/gluon-ts
Python, PyTorch PyTorchTS Clone of GluonTS in PyTorch. https://github.com/zalandoresearch/pytorch-ts
§2.7.10. Machine learning R RSNNS mlp; rbf; dlvq; elman; jordan; som Neural Networks using the Stuttgart Neural Network Simulator (SNNS). https://cran.r-project.org/package=RSNNS
§2.7.10. Machine learning (continued) R rpart rpart; prune Recursive partitioning and regression trees. https://cran.r-project.org/package=rpart
R caret Classification and regression training. https://cran.r-project.org/package=caret
R e1071 svm Misc ML functions of the Department of Statistics, Probability Theory Group. https://cran.r-project.org/package=e1071
R kernlab gausspr Gaussian processes for regression and classification. https://cran.r-project.org/package=kernlab
R brnn brnn Bayesian Regularisation for Feed-Forward Neural Networks. https://cran.r-project.org/package=brnn
R grnn grnn General regression neural network. https://cran.r-project.org/package=grnn
R randomForest randomForest Breiman and Cutler’s Random Forests for Classification and Regression. https://cran.r-project.org/package=randomForest
R gbm gbm Generalised Boosted regression models. https://cran.r-project.org/package=gbm
R neuralnet neuralnet Training of simple Neural Networks. https://cran.r-project.org/package=neuralnet
Python Tensorflow A framework, developed by Google, offering tools for designing, building, and deploying ML models. https://tensorflow.org/api_docs/python/tf
Python Keras API A deep learning API built on top of Tensorflow. It provides high level blocks for building and training NN models. https://keras.io/
R Tensorflow R Interface to Tensorflow (https://tensorflow.rstudio.com/). https://tensorflow.rstudio.com/
R deepnet Deep learning toolkit. https://cran.r-project.org/package=deepnet
R h2o R Interface for the ‘H2O’ Scalable Machine Learning Platform. https://cran.r-project.org/package=h2o
R Apache MXNet A flexible library for deep learning. https://mxnet.apache.org/
Python scikit-learn Ordinary Least Squares, Ridge regression, Lasso, Bayesian Regression, Generalized Linear Regression, Stochastic Gradient Descent and Polynomial regression, Support Vector Machines, Nearest Neighbors, Gaussian Processes, Decision Trees, Ensemble methods (Forests of randomised trees, AdaBoost and Gradient Tree Boosting), Multi-layer Perceptrons. https://scikit-learn.org/stable/
Python CNTK A framework, developed by Microsoft, that provides tools for building ML and DL models. https://docs.microsoft.com/en-us/cognitive-toolkit/
Python PyTorch A framework, developed by Facebook, for building ML and DL models. https://pytorch.org/
§2.7.12. Clustering-based forecasting R tsfknn tsfknn Time Series Forecasting Using Nearest Neighbours. https://cran.r-project.org/package=tsfknn
§2.7.13. Hybrid methods Python ESRNN-GPU A GPU-enabled version of the hybrid model used by the winner of M4 competition. https://github.com/damitkwr/ESRNN-GPU
§2.8.1. Parametric methods for intermittent demand forecasting R tsintermittent crost; tsb Parametric forecasting methods for intermitternt demand. https://cran.r-project.org/package=tsintermittent
R forecast croston Forecasts for intermittent demand using Croston’s method. https://cran.r-project.org/package=forecast
§2.8.2. Non-parametric intermittent demand methods R tsintermittent imapa MAPA for intermittent demand data. https://cran.r-project.org/package=tsintermittent
§2.8.3. Classification methods R tsintermittent idclass Time series categorisation for intermittent demand. https://cran.r-project.org/package=tsintermittent
R tsutils abc; xyz; abcxyz Classification functions and routines. https://cran.r-project.org/package=tsutils
§2.9.3. Forecasting with text information Python NLTK The Natural Language Toolkit in Python. https://www.nltk.org/
Python SpaCy An open source library for advanced Natural Language Processing in Python. https://spacy.io/
§2.10.1. Cross-sectional hierarchical forecasting R hts Functions and routines for hierarchical and grouped time series forecasting. https://cran.r-project.org/package=hts
§2.10.2. Temporal aggregation R MAPA mapa; mapasimple Functions and wrappers for using the Multiple Aggregation Prediction Algorithm (MAPA) for time series forecasting. https://cran.r-project.org/package=MAPA
R thief thief Temporal Hierarchical Forecasting. https://cran.r-project.org/package=thief
R tsintermittent imapa MAPA for intermittent demand data with automatic model selection based on the PK classification. https://cran.r-project.org/package=tsintermittent
§2.10.4. Ecological inference forecasting R ei ei Returns local and global forecasts of inner cells in \(2 \times 2\) tables. https://cran.r-project.org/package=ei
R eiPack ei.MD.bayes; ei.reg; ei.reg.bayes Returns local and global forecasts of inner cells in R\(\times\)C tables under a Multinomial Dirichlet model or using ecological regression. https://cran.r-project.org/package=eiPack
R lphom lphom; tslphom; nslphom Returns forecasts of inner cells of a R\(\times\)C table using linear programming optimisation. https://CRAN.R-project.org/package=lphom
R eiCompare ei_est_gen; ei_good; ei_rxc Returns forecasts of inner cells of a \(R \times C\) tables using iterative versions of \(2 \times 2\) methods and the Multinomial Dirichlet model. https://cran.r-project.org/package=eiCompare
§2.12.2. Point, interval, and pHDR forecast error measures R forecast accuracy Accuracy measures for a forecast model. https://cran.r-project.org/package=forecast
§2.12.4. Evaluating probabilistic forecasts R scoringRules Scoring rules for parametric and simulated distribution forecasts. https://cran.r-project.org/package=scoringRules
R verification crps Continuous ranked probability score. https://cran.r-project.org/package=verification
§2.12.6. Statistical tests of forecast performance R forecast dm.test Diebold-Mariano test for predictive accuracy. https://cran.r-project.org/package=forecast
R tsutils nemenyi Nonparametric multiple comparisons (Nemenyi test). https://cran.r-project.org/package=tsutils
§2.12.6. Statistical tests of forecast performance (continued) Gretl FEP doMZtest; doHPtest; doEKTtest; doPTtest; doDLtest; doDMtest; doGWtest; doCWtest Various statistical tests on forecast unbiasedness, efficiency, asymmetric loss and directional changes. http://gretl.sourceforge.net/
Gretl DiebMar Diebold-Mariano test. http://gretl.sourceforge.net/