Sign Up for the Quarterly Newsletter

HCIBench

HCIBench stands for "Hyper-converged Infrastructure Benchmark". It's essentially an automation wrapper around the popular and proven open source benchmark tools: Vdbench and Fio that make it easier to automate testing across a HCI cluster. HCIBench aims to simplify and accelerate customer POC performance testing in a consistent and controlled way. The tool fully automates the end-to-end process of deploying test VMs, coordinating workload runs, aggregating test results, performance analysis and collecting necessary data for troubleshooting purposes.

HCIBench is not only a benchmark tool designed for vSAN, but also could be used to evaluate the performance of all kinds of Hyper-Converged Infrastructure Storage in vSphere environment.

  • Web Browser:
    IE8+, Firefox or Chrome
  • vSphere 5.5 and later environments for both HCIBench and its client VMs deployment
See the HCIBench_2.5_User_Guide.pdf for details.

Version 2.5.1 Update

  • Fixed non vSAN cluster testing issue
  • Fixed pre-validation storage policy issue
  • Added in-place upgrading capability, to do so, you can also upgrade HCIBench to the latest version by running the following command in HCIBench: tdnf install -y git && git clone https://github.com/cwei44/HCIBench.git && sh HCIBench/upgrade.sh
    MD5 Checksum: 1d14426f92b353e90469a8623ade2bc1 HCIBench_2.5.1.ova
 

Version 2.5.0 Update

  • Added support vSAN HCI Mesh testing, now you can test both local and remote vSAN datastores at the same time

  • Added support local storage including VMFS and vSAN-Direct testing

  • Added vSAN Debug Mode, allow user to collect vm-support bundle and vmkstats automatically when running testing against vSAN

  • Changed guest VMs name convention to {vm_prefix}-{datastore_id}-batch_num-sequence_num

  • Enhanced testing report format

  • Allow user to specify customized IP addresses for guest VMs

  • Allow user to configure CPU and Memory for guest VMs

  • Added best practice and network troubleshooting guide in the user manual

  • Bug fixes

  • MD5 Checksum: 817c2c788364f252e728d4253b3b96da HCIBench_2.5.0.ova

Version 2.4.0 Update

  • Fixed tvm deployment bug when specifying host
  • enabled easy run to support stretched cluster
  • fixed timezone issue on pdf report, and added more vSAN info into PDF report
  • set testname and testcase as variables in grafana
  • added CPU workload into fio config page
  • updated rbvmomi to support vsphere 7.0+
  • enhanced fio and vdbench graphite dashboards
    MD5 Checksum: 0cfd6cc852e33e5ce32022a66539b4c9 HCIBench_2.4.0.ova

Version 2.3.1 Update

  • Fixed static IP setting issue
  • Fixed reuse VMs on multi datastores issue
  • Fixed vm/tvm deployment issue
  • MD5 Checksum: 1b220f22575eacf62a965992a4c916e7 HCIBench_2.3.1.ova

Version 2.3.0 Update

Version 2.2.1 Update

  • Fixed docker volume moving issue
  • MD5 checksum of HCIBench_2.2.1.ova: 1a39c9df7d1485bc06332ae0b9d92ca7

Version 2.2 Update

  • Moved docker volume to sdb to avoid blowing up OS disk
  • Added Fio spreadsheet generator
  • Added DRS warning checkup 
  • Enhanced Grafana to keep all the historical data 
  • Added DNS exception handler 
  • Fixed RAM and PCPU reporting issue 
  • Fixed Vdbench spreadsheet not reporting issue 
  • MD5 checksum of HCIBench_2.2.ova: bb2a77dcf2ecc23b1ec2c30aee9945ec

Version 2.1 Update

  • Switched UI to dark theme 
  • Redesigned VMDK preparation methodology, which can complete much faster using RANDOM on deduped storage 
  • Added VMDK preparation process update 
  • Added Graphite port check into prevalidation 
  • Added vCenter/Host password obfuscation 
  • Added "Delete Guest VM" button 
  • Fixed Grafana display issue 
  • Fixed FIO blank results issue 
  • Bug fixes 
  • MD5 checksum of HCIBench_2.1.ova: d37e6f164ed962a6e7ccbe104ba9eaec

Version 2.0 Update

  • Added fio as an alternative workload generator
  • Added Grafana for workload live monitoring
  • Switched UI to clarity
  • Allow user to select one to four cases while using easy-run
  • Bug fixes
  • MD5 checksum of HCIBench_2.0.ova: ba3c2b06b8c27fb41a1bb68baedb325f 

Version 1.6.8.7 Update

  • Enhanced easy-run, put original 4k,70% read as the first test case, then 4k, 100% read and 256k, 100% write
  • Enhanced tvm deployment validation
  • Added Checksum into easy-run consideration
  • Updated guest VM template with increased ring_pages and disk scheduler
  • Added DNS configuration guidance into welcome message

Version 1.6.8.5 Update

  • Added 2 more test cases into easy-run, 4k 100% random read and 256k 100% sequential write
  • Batch deployment will be involved if deploying more than 8 VMs to speed up deployment process
  • Allow user to choose IP prefix when using static IP
  • Optimized UI to allow user to review the results by single click
  • Fixed regression issue when placing Datacenter/Cluster in the folder

Version 1.6.8.1 Update

  • Fixed regression when datastore is in the datastore folder
  • Avoid checking connection to host directly and use tvm deployment instead
  • Added Vdbench version check in summary script

Version 1.6.8 Update

  • Added resource pool and VM folder fields for VMC environment
  • Fixed easy-run disk size issue
  • Enhanced pre-validation error message handling
  • Changed the names of network interface from "Public Network" to "Management Network", and "Private Network" to "VM Network"

Version 1.6.7.2 Update

  • Enhanced write/read buffer/cache methodology
  • Fixed network ip-prefix selection issue
  • Fixed 95% percentile calculation issue

Version 1.6.7.1 Update

  • Fixed vSAN Performance Diagnostic API call
  • Fixed network validation message not clear issue
  • Fixed setting re-use VMs as default bug in 1.6.7

Version 1.6.7 Update

  • Enabled https instead of http
  • Added storage policy field, user can specify storage policy for the data disks. For this version, storage policy can't be assigned to existing client VMs
  • Enhanced deployment methodology
  • Enhanced vSAN Observer to avoid blow up the memory
  • Enhanced vSAN Performance Diagnostic API call with HCIBench workload configuration included
  • Added timestamp to the testing status
  • Bug fixes

Version 1.6.6. Update

  • Spectre & Meldown patch on both HCIBench VM and Client VM
  • Added client VM prefix field, allow running multiple HCIBench instances against single cluster
  • Attach testing log along with testing results
  • Enabled live vSAN Observer when running testing, using https://HCIBench_IP:8010
  • Updated the drop read/write cache script
  • Added more message info during the testing
  • Bug fixes

Version 1.6.5.2 Update

  • Added case comparisons by generating an XLS file for each test folder
  • Fixed bug when there's white space in datastore name or test name

Version 1.6.5.1 Update

  • Enhanced IP segment selection
  • Set open file limit to 4096
  • Updated vm-tools to the latest version
  • Bug fixes

Version 1.6.5 Update

  • Enhanced 95th percentile calculation.
  • Added Curve and Multi Run calculation.
  • Added SSH Service validation.
  • Replaced DHCP Service with Static IP Service.
  • Added IP conflict check.
  • Fixed bunch of bugs.
  • Change the default client VM RAM from 4GB to 8GB

Version Version 1.6.3 Update

  • Enhanced vSANPerformanceDiagnose function call
  • Enhanced port 443 validation
  • Enhanced results calculation
  • Added host maintenance mode validation
  • Added deployment validation

Version 1.6.2 Update

  • Integrated with vSAN Performance Diagnostic of vSphere_6.5U1/vSAN_6.6.1.
  • Added DHCP Service validation.
  • Added Vdbench workload profile validation.
  • Removed the root password expiration policy.
  • Changed results display to show full file names.
  • Changed easy-run calculation from host basis to disk-group basis.

Version 1.6.1 Update

  • Added network name uniqueness check
  • Changed the "disk warmup" to "Virtual disk preparation" to avoid confusion
  • Changed the pvscsi configuration, when there are more than 4 vmdks per pvscsi, more pvscsi controllers will be added and the vmdks will be evenly distributed
  • Bug fixes

Version 1.6.0.0 Update

  • Added Clear read/write cache option for vSAN.
  • Added Easy Run feature for vSAN, Easy Run can help vSAN user to determine the VMs/VMDKs/size/Disk_Init_Methods and run testing automatically.
  • Added Re-use VMs feature, user is able to re-use the existing client VMs for more tests.
  • Added the 95th percentile of calculation into the results.
  • Had special characters issue resolved and got other bug fixes.

Version 1.5.0.5 Update

  • Increased Timeout value of client VM disk from 30 seconds to 180 seconds.
  • Disabled client VM password expiration.
  • Disabled client VM OS disk fsck.
  • Set Observer interval to 60 seconds to shrink the size of observer data.
  • Fixed PCPU calculation.
  • Created link directory of /opt/automation/logs, user will be able to review the testing logs in http://HCIBENCH/hcibench_logs/
  • Increased the RAM of HCIBench from 4GB to 8GB to avoid running out-of-resource issue.

Version 1.5.0.4 Update

  • Added the checking if user saved the configuration
  • Increased the stack size to 65536 due to "ls too many arguments issue while process PCPU usage"
  • Fixed the typo from "Netowrk" -> "Network" in the deployment page.
  • Added vCenter hostname resolve checking.
  • Fixed the client VM OS VMDK size from 15.5GB to 16GB; changed the vmdk size specification from decimal format to binary format.

Version 1.5.0.3 Update

  • The bug "When tested against a non-vSAN datastore and having 'Directly Deploy on Hosts' checked, the test won't go through after deployment." is fixed.
  • Enhancement: extract storage policy information when doing pre-validation and checks if the deployment size would be too aggressive to vSAN.

Version 1.5.0.2 Update

  • Bug "test would fail if datastores' name contains white space" fixed.