Performance Visualizations for the bbob
test suite
Below, we provide postprocessed data showing the performance of all 250+ officially supported algorithm data sets for the bbob
test suite.
Due to the large amount of algorithms (and the limited space in the figures), we currently group algorithm data sets by year of publication.
Performance Comparisons per Year
2009: ALPS, AMALGAM, BAYEDA, BFGS, BIPOP-CMA-ES, CMA-ESPLUSSEL, Cauchy-EDA, DASA, DE-PSO, DIRECT, EDA-PSO, FULLNEWUOA, G3PCX, GA, GLOBAL, IPOP-SEP-CMA-ES, LSfminbnd, LSstep, MA-LS-CHAIN, MCS, NELDERDOERR, NELDER, NEWUOA, ONEFIFTH, POEMS, PSO, PSO, RANDOMSEARCH, Rosenbrock, VNS, iAMALGAM
2010: 1komma2, 1komma2mir, 1komma2mirser, 1komma2ser, 1komma4, 1komma4mir, 1komma4mirser, 1komma4ser, 1plus1, 1plus2mirser, ABC, AVGNEWUOA, CMAEGS, DE-F-AUC, DEuniform, IPOP-ACTCMA-ES, IPOP-CMA-ES, MOS, NBC-CMA, PM-AdapSS-DE, RCGA, SPSA, oPOEMS, pPOEMS
2012: ACOR, BIPOPaCMA, BIPOPsaACM, CMAES, CMA, CMAa, CMAm, CMAma, CMAmah, CMAmh, DBRCGA, DE-AUTO, DE-BFGS, DE-ROLL, DE-SIMPLEX, DEAE, DE, DEb, DEctpb, IPOPsaACM, JADE, JADEb, JADEctpb, MVDE, NBIPOPaCMA, NIPOPaCMA, PSO-BFGS, SNES, xNES, xNESas
2013: BIPOP-aCMA-STEP, BIPOP-saACM-k, CGA-grid100, CGA-grid16, CGA-ring100, CGA-ring16, CMAES, DE, GA-100, HCMA, HILL, HMLSL, IP-10DDr, IP-500, IPOP400D, IP, MEMPSODE, MLSL, OQNLP, P-DCN, P-zero, PRCGA, SMAC-BBOB, U-DCN, U-zero, fmincon, fminunc, lmm-CMA-ES, simplex, tany, texp, GP1-CMAES, GP5-CMAES, IPOPCMAv3p61, RF1-CMAES, RF5-CMAES
2014: BFGS-scipy, CG-Fletcher-Reeves-scipy, CMA-ES-python, EG50-cocopf, L-BFGS-B-scipy, Nelder-Mead-scipy, Powell-scipy, SLSQP-scipy, UNIF-cocopf
2015: MATSUMOTO, R-DE-10e2, R-DE-10e5, R-SHADE-10e2, R-SHADE-10e5, RL-SHADE-10e2, RL-SHADE-10e5, SOO, BrentSTEPif, BrentSTEPifeg, BrentSTEPqi, BrentSTEPrr, CMA-CSA, CMA-MSR, CMA-TPA, GP1-CMAES, GP1-CMAES, GP5-CMAES, GP5-CMAES, IPOPCMAv3p61, IPOPCMAv3p61, LHD-10xDefault-MATSuMoTo, LHD-2xDefault-MATSuMoTo, RAND-2xDefault-MATSuMoTo, RF1-CMAES, RF1-CMAES, RF5-CMAES, RF5-CMAES, STEPif, STEPifeg, STEPrr
2016: PSAaLmC-CMA-ES, PSAaLmD-CMA-ES, PSAaSmC-CMA-ES, PSAaSmD-CMA-ES
2017: CMAES-APOP, DTS-CMA-ES, EvoSpace-PSO-GA, KL-BIPOP-CMA-ES, KL-IPOP-CMA-ES, KL-Restart-CMA-ES, Ord-H-DTS-CMA-ES, Ord-N-DTS-CMA-ES, Ord-Q-DTS-CMA-ES, SSEABC, RANDOMSEARCH-4-1e7D, RANDOMSEARCH-4p5-1e7D, RANDOMSEARCH-5-1e7D
2018: BBDE-N, BBDE-best, BBDE-ttb, BBDE, BFGS-M-17, BFGS-P-09, BFGS-P-Instances, BFGS-P-StPt, BFGS-P-range, CMAES-APOP-Var1, CMAES-APOP-Var2, CMAES-APOP-Var3, DE-best, DE-rand, DE-ttb, PSA-CMA-ES, PSA-CMA-ESwRS, DTS-CMA-ES, uBBDE-N, uBBDE-best, uBBDE-ttb, uBBDE, uDE-best, uDE-rand, uDE-ttb
2019: Adaptive-Two-Mode, BFGS-scipy-2019, CG-scipy-2019, COBYLA-scipy-2019, DE-scipy-2019, GNN-CMA-ES, IPOP-CMA-ES-2019, L-BFGS-B-scipy-2019, Nelder-Mead-scipy-2019, Powell-scipy-2019, RS-0p5-initIn0, RS-0p5, RS-1-initIn0, RS-10-initIn0, RS-10, RS-1, RS-2-initIn0, RS-20-initIn0, RS-20, RS-2, RS-3-initIn0, RS-3, RS-4-initIn0, RS-5-initIn0, RS-6-initIn0, RS-6, SLSQP-scipy-2019, TNC-scipy-2019, adapt-Nelder-Mead-scipy-2019, CMA-ES-2019
2020: CMA-ES-2019, HE-ES, SLSQP+lq-CMA-ES, SLSQP-11-scipy, lq-CMA-ES
2021: SHADE-LM-POP4-to-10, SHADE-LM
2022: CMA-ES-Akimoto, CMA-ES-pycma, CMAES-APOP-KMA, CMAES-APOP-KP, CMAES-APOP-MA, DD-CMA-ES-Akimoto, DD-CMA-ES-pycma, EXPLO2, oMads-2N, oMads-Neg
2023: BIRMIN, DIRECT-REV, E-WOA, I-DBDP-GL, WOA, a-CMA-ES, ad-CMA-ES, adm-CMA-ES, c-CMA-ES, cd-CMA-ES, cdm-CMA-ES, cma-bp-bbob, cma-bt-bbob, d-CMA-ES, default-CMA-ES, dm-CMA-ES, m-SDWOA, s-CMA-ES, sc-CMA-ES, scd-CMA-ES, scdm-CMA-ES, sd-CMA-ES, sdm-CMA-ES
Example code to produce the figures
The Python code to locally generate the second entry, 2010, above (other entries work respectively) reads
import cocopp # see https://pypi.org/project/cocopp
= {None: cocopp.archives.bbob.get_all('2009')}
cocopp.genericsettings.background 'bbob/2010/*') # will take several minutes to process cocopp.main(