Sign Up for the Quarterly Newsletter

HCIBench

HCIBench stands for "Hyper-converged Infrastructure Benchmark". It's essentially an automation wrapper around the popular and proven open source benchmark tools: Vdbench and Fio that make it easier to automate testing across a HCI cluster. HCIBench aims to simplify and accelerate customer POC performance testing in a consistent and controlled way. The tool fully automates the end-to-end process of deploying test VMs, coordinating workload runs, aggregating test results, performance analysis and collecting necessary data for troubleshooting purposes.

HCIBench is not only a benchmark tool designed for vSAN, but also could be used to evaluate the performance of all kinds of Hyper-Converged Infrastructure Storage in vSphere environment.

  • Web Browser:
    IE8+, Firefox or Chrome
  • vSphere 5.5 and later environments for both HCIBench and its client VMs deployment
See the HCIBench_2.0_User_Guide.pdf for details.

Version 2.4.0

  1. Fixed tvm deployment bug when specifying host
  2. enabled easy run to support stretched cluster
  3. fixed timezone issue on pdf report, and added more vSAN info into PDF report
  4. set testname and testcase as variables in grafana
  5. added CPU workload into fio config page
  6. updated rbvmomi to support vsphere 7.0+
  7. enhanced fio and vdbench graphite dashboards
    MD5 Checksum: 0cfd6cc852e33e5ce32022a66539b4c9 HCIBench_2.4.0.ova

Version 2.3.1

  • Fixed static IP setting issue
  • Fixed reuse VMs on multi datastores issue
  • Fixed vm/tvm deployment issue
  • MD5 Checksum: 1b220f22575eacf62a965992a4c916e7 HCIBench_2.3.1.ova

Version 2.3.0

Version 2.2.1

  • Fixed docker volume moving issue
  • MD5 checksum of HCIBench_2.2.1.ova: 1a39c9df7d1485bc06332ae0b9d92ca7

Version 2.2

  • Moved docker volume to sdb to avoid blowing up OS disk
  • Added Fio spreadsheet generator
  • Added DRS warning checkup 
  • Enhanced Grafana to keep all the historical data 
  • Added DNS exception handler 
  • Fixed RAM and PCPU reporting issue 
  • Fixed Vdbench spreadsheet not reporting issue 
  • MD5 checksum of HCIBench_2.2.ova: bb2a77dcf2ecc23b1ec2c30aee9945ec

Version 2.1

  • Switched UI to dark theme 
  • Redesigned VMDK preparation methodology, which can complete much faster using RANDOM on deduped storage 
  • Added VMDK preparation process update 
  • Added Graphite port check into prevalidation 
  • Added vCenter/Host password obfuscation 
  • Added "Delete Guest VM" button 
  • Fixed Grafana display issue 
  • Fixed FIO blank results issue 
  • Bug fixes 
  • MD5 checksum of HCIBench_2.1.ova: d37e6f164ed962a6e7ccbe104ba9eaec

Version 2.0

  • Added fio as an alternative workload generator
  • Added Grafana for workload live monitoring
  • Switched UI to clarity
  • Allow user to select one to four cases while using easy-run
  • Bug fixes
  • MD5 checksum of HCIBench_2.0.ova: ba3c2b06b8c27fb41a1bb68baedb325f 

Version 1.6.8.7

  • Enhanced easy-run, put original 4k,70% read as the first test case, then 4k, 100% read and 256k, 100% write
  • Enhanced tvm deployment validation
  • Added Checksum into easy-run consideration
  • Updated guest VM template with increased ring_pages and disk scheduler
  • Added DNS configuration guidance into welcome message

Version 1.6.8.5

  • Added 2 more test cases into easy-run, 4k 100% random read and 256k 100% sequential write
  • Batch deployment will be involved if deploying more than 8 VMs to speed up deployment process
  • Allow user to choose IP prefix when using static IP
  • Optimized UI to allow user to review the results by single click
  • Fixed regression issue when placing Datacenter/Cluster in the folder

Version 1.6.8.1

  • Fixed regression when datastore is in the datastore folder
  • Avoid checking connection to host directly and use tvm deployment instead
  • Added Vdbench version check in summary script

Version 1.6.8

  • Added resource pool and VM folder fields for VMC environment
  • Fixed easy-run disk size issue
  • Enhanced pre-validation error message handling
  • Changed the names of network interface from "Public Network" to "Management Network", and "Private Network" to "VM Network"

Version 1.6.7.2

  • Enhanced write/read buffer/cache methodology
  • Fixed network ip-prefix selection issue
  • Fixed 95% percentile calculation issue

Version 1.6.7.1

  • Fixed vSAN Performance Diagnostic API call
  • Fixed network validation message not clear issue
  • Fixed setting re-use VMs as default bug in 1.6.7

Version 1.6.7

  • Enabled https instead of http
  • Added storage policy field, user can specify storage policy for the data disks. For this version, storage policy can't be assigned to existing client VMs
  • Enhanced deployment methodology
  • Enhanced vSAN Observer to avoid blow up the memory
  • Enhanced vSAN Performance Diagnostic API call with HCIBench workload configuration included
  • Added timestamp to the testing status
  • Bug fixes

Version 1.6.6.

  • Spectre & Meldown patch on both HCIBench VM and Client VM
  • Added client VM prefix field, allow running multiple HCIBench instances against single cluster
  • Attach testing log along with testing results
  • Enabled live vSAN Observer when running testing, using https://HCIBench_IP:8010
  • Updated the drop read/write cache script
  • Added more message info during the testing
  • Bug fixes

Version 1.6.5.2

  • Added case comparisons by generating an XLS file for each test folder
  • Fixed bug when there's white space in datastore name or test name

Version 1.6.5.1

  • Enhanced IP segment selection
  • Set open file limit to 4096
  • Updated vm-tools to the latest version
  • Bug fixes

Version 1.6.5

  • Enhanced 95th percentile calculation.
  • Added Curve and Multi Run calculation.
  • Added SSH Service validation.
  • Replaced DHCP Service with Static IP Service.
  • Added IP conflict check.
  • Fixed bunch of bugs.
  • Change the default client VM RAM from 4GB to 8GB

Version Version 1.6.3

  • Enhanced vSANPerformanceDiagnose function call
  • Enhanced port 443 validation
  • Enhanced results calculation
  • Added host maintenance mode validation
  • Added deployment validation

Version 1.6.2

  • Integrated with vSAN Performance Diagnostic of vSphere_6.5U1/vSAN_6.6.1.
  • Added DHCP Service validation.
  • Added Vdbench workload profile validation.
  • Removed the root password expiration policy.
  • Changed results display to show full file names.
  • Changed easy-run calculation from host basis to disk-group basis.

Version 1.6.1

  • Added network name uniqueness check
  • Changed the "disk warmup" to "Virtual disk preparation" to avoid confusion
  • Changed the pvscsi configuration, when there are more than 4 vmdks per pvscsi, more pvscsi controllers will be added and the vmdks will be evenly distributed
  • Bug fixes

Version 1.6.0.0

  • Added Clear read/write cache option for vSAN.
  • Added Easy Run feature for vSAN, Easy Run can help vSAN user to determine the VMs/VMDKs/size/Disk_Init_Methods and run testing automatically.
  • Added Re-use VMs feature, user is able to re-use the existing client VMs for more tests.
  • Added the 95th percentile of calculation into the results.
  • Had special characters issue resolved and got other bug fixes.

Version 1.5.0.5

  • Increased Timeout value of client VM disk from 30 seconds to 180 seconds.
  • Disabled client VM password expiration.
  • Disabled client VM OS disk fsck.
  • Set Observer interval to 60 seconds to shrink the size of observer data.
  • Fixed PCPU calculation.
  • Created link directory of /opt/automation/logs, user will be able to review the testing logs in http://HCIBENCH/hcibench_logs/
  • Increased the RAM of HCIBench from 4GB to 8GB to avoid running out-of-resource issue.
  • Version 1.5.0.4

    • Added the checking if user saved the configuration
    • Increased the stack size to 65536 due to "ls too many arguments issue while process PCPU usage"
    • Fixed the typo from "Netowrk" -> "Network" in the deployment page.
    • Added vCenter hostname resolve checking.
    • Fixed the client VM OS VMDK size from 15.5GB to 16GB; changed the vmdk size specification from decimal format to binary format.

    Version 1.5.0.3

    • The bug "When tested against a non-vSAN datastore and having 'Directly Deploy on Hosts' checked, the test won't go through after deployment." is fixed.
    • Enhancement: extract storage policy information when doing pre-validation and checks if the deployment size would be too aggressive to vSAN.

    Version 1.5.0.2

    • Bug "test would fail if datastores' name contains white space" fixed.