From 4cfaa9809b815638fc99388f3c485616520bf950 Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Wed, 5 Feb 2020 16:54:59 -0800 Subject: [PATCH 1/6] Update readme with more info on how to run spec tests --- test/vagrant-spec/readme.md | 56 ++++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 16 deletions(-) diff --git a/test/vagrant-spec/readme.md b/test/vagrant-spec/readme.md index dbbc920e8..d9ac5e04a 100644 --- a/test/vagrant-spec/readme.md +++ b/test/vagrant-spec/readme.md @@ -1,5 +1,11 @@ # Running vagrant-spec +The vagrant-spec project is where Vagrant acceptance tests live and are run. +__NOTE:__ You must use a hypervisor that allows for nested virtualization to run these tests. +So for the _vagrant_ project, it uses the vagrant vmware plugin as a host. If you +want to test this locally, please keep in mind that you will need this hypervisor +to properly run the tests locally. + ## Requirements - vagrant installed (from source, or from packages) @@ -7,9 +13,28 @@ - ![vagrant](https://github.com/hashicorp/vagrant) repo - ![vagrant-spec](https://github.com/hashicorp/vagrant-spec) repo +## Relevant environment variables: + +Below are some environment variables used for running vagrant-spec. Many of +these are required for defining which hosts and guests to run the tests on. + +- VAGRANT_CLOUD_TOKEN + + Token to use if fetching a private box (like windows). This does not have to be explicitly + set if you log into Vagrant cloud with `vagrant cloud login`. +- VAGRANT_HOST_BOXES + - Vagrant box to use as a host for installing VirtualBox and bringing up Vagrant guests to test +- VAGRANT_GUEST_BOXES + - Vagrant box to use as a guest to run tests on +- VAGRANT_CWD + - Directory location of vagrant-spec Vagrantfile inside of the Vagrant source repo +- VAGRANT_VAGRANTFILE + - Vagrantfile to use for running vagrant-spec. Unless changed, this should be set as `Vagrantfile.spec`. +- VAGRANT_HOST_MEMORY + - Set how much memory your host will use (defaults to 2048) + ## How to run -First, we need to build vagrant-spec: +First, we need to build vagrant-spec and copy the built gem into the Vagrant source repo: ``` cd vagrant-spec @@ -18,31 +43,30 @@ cp vagrant-spec-0.0.1.gem /path/to/vagrant/vagrant-spec.gem ``` Next, make a decision as to which host and guest boxes will be used to run the tests. +A list of valid hosts and guests can be found in the `Vagrantfile.spec` adjacent +to this readme. From the root dir of the `vagrant` project, run the following command: ```shell -VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up +VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up --provider vmware_desktop ``` If you are running windows, you must give your host box more memory than the default. That can be done through the environment variable `VAGRANT_HOST_MEMORY` ```shell -VAGRANT_HOST_MEMORY=10000 VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up +VAGRANT_HOST_MEMORY=10000 VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up --provider vmware_desktop ``` +__Note:__ It is not required that you invoke Vagrant directly in the source repo, so +if you wish to run it else where, be sure to properly set the `VAGRANT_CWD` environment +variable to point to the proper test directory inside of the Vagrant source. -## Relevant environment variables: +### About Vagrantfile.spec -- VAGRANT_CLOUD_TOKEN - + Token to use if fetching a private box (like windows) -- VAGRANT_HOST_BOXES - - Vagrant box to use to host and run tests -- VAGRANT_GUEST_BOXES - - Vagrant box to use to run tests on -- VAGRANT_CWD - - Directory location of vagrant-spec Vagrantfile -- VAGRANT_VAGRANTFILE - - Vagrantfile to use for running vagrant-spec -- VAGRANT_HOST_MEMORY - - Set how much memory your host will use (defaults to 2048) +This Vagrantfile expects the box used to end in a specific "platform", so that it can associate +a provision script with the correct plaform. Because some boxes might not end in +their platform (like `hashicorp-vagrant/ubuntu-16.04` versus `hashicorp/bionic64`, +there is a hash defined called `PLATFORM_SCRIPT_MAPPING` that will tell vagrant +which platform script to provision with rather than relying on the box ending with +the name of the platform. From 9aaea362c6895ad23caf0b11b22f24536c574e87 Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Wed, 5 Feb 2020 16:55:13 -0800 Subject: [PATCH 2/6] Introduce platform mapping for other kinds of vagrant boxes --- test/vagrant-spec/Vagrantfile.spec | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/test/vagrant-spec/Vagrantfile.spec b/test/vagrant-spec/Vagrantfile.spec index 1e8fbc34f..8188f1338 100644 --- a/test/vagrant-spec/Vagrantfile.spec +++ b/test/vagrant-spec/Vagrantfile.spec @@ -1,5 +1,9 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + # Guest boxes to use for vagrant-spec GUEST_BOXES = { + 'hashicorp/bionic64' => '1.0.282', 'hashicorp-vagrant/ubuntu-16.04' => '1.0.1', 'hashicorp-vagrant/centos-7.4' => '1.0.2', 'hashicorp-vagrant/windows-10' => '1.0.0', @@ -8,12 +12,22 @@ GUEST_BOXES = { # Host boxes to run vagrant-spec HOST_BOXES = { + 'hashicorp/bionic64' => '1.0.282', 'hashicorp-vagrant/ubuntu-16.04' => '1.0.1', 'hashicorp-vagrant/centos-7.4' => '1.0.2', 'hashicorp-vagrant/windows-10' => '1.0.0', 'spox/osx-10.12' => '0.0.1' } +# Not all boxes are named by their specific "platform" +# so this allows Vagrant to use the right provision script +PLATFORM_SCRIPT_MAPPING = { + "ubuntu" => "ubuntu", + "bionic" => "ubuntu", + "centos" => "centos", + "windows" => "windows" +} + # Determine what providers to test enabled_providers = ENV.fetch("VAGRANT_SPEC_PROVIDERS", "virtualbox").split(",") # Set what boxes should be used @@ -63,9 +77,11 @@ Vagrant.configure(2) do |global_config| vmware.vmx['vhv.allow'] = 'TRUE' end if platform == "windows" - config.vm.provision :shell, path: "./scripts/#{platform}-setup.#{provider_name}.ps1", run: "once" + config.vm.provision :shell, + path: "./scripts/#{PLATFORM_SCRIPT_MAPPING[platform]}-setup.#{provider_name}.ps1", run: "once" else - config.vm.provision :shell, path: "./scripts/#{platform}-setup.#{provider_name}.sh", run: "once" + config.vm.provision :shell, + path: "./scripts/#{PLATFORM_SCRIPT_MAPPING[platform]}-setup.#{provider_name}.sh", run: "once" end guest_boxes.each_with_index do |box_info, idx| guest_box, box_version = box_info @@ -86,7 +102,7 @@ Vagrant.configure(2) do |global_config| else config.vm.provision( :shell, - path: "./scripts/#{platform}-run.#{provider_name}.sh", + path: "./scripts/#{PLATFORM_SCRIPT_MAPPING[platform]}-run.#{provider_name}.sh", keep_color: true, env: { "VAGRANT_SPEC_ARGS" => "--no-builtin #{spec_cmd_args}".strip, From ef2075c9b265b69a27226c281722ee63c6c81ba9 Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Thu, 6 Feb 2020 14:46:11 -0800 Subject: [PATCH 3/6] Ensure nfs kernel server is on ubuntu hosts --- test/vagrant-spec/scripts/ubuntu-setup.virtualbox.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/test/vagrant-spec/scripts/ubuntu-setup.virtualbox.sh b/test/vagrant-spec/scripts/ubuntu-setup.virtualbox.sh index aac4bed9e..a8a8d32b7 100644 --- a/test/vagrant-spec/scripts/ubuntu-setup.virtualbox.sh +++ b/test/vagrant-spec/scripts/ubuntu-setup.virtualbox.sh @@ -4,6 +4,7 @@ set -xe apt-get update -q apt-get install -qy linux-headers-$(uname -r) apt-get install -qy virtualbox +apt-get install -qy nfs-kernel-server pushd /vagrant From 8dc45976d655ce8ec6f639a0c74e34a06a7e6e69 Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Fri, 7 Feb 2020 13:29:36 -0800 Subject: [PATCH 4/6] Document VAGRANT_SPEC_ARGS option for running tests --- test/vagrant-spec/readme.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/test/vagrant-spec/readme.md b/test/vagrant-spec/readme.md index d9ac5e04a..1e26bb281 100644 --- a/test/vagrant-spec/readme.md +++ b/test/vagrant-spec/readme.md @@ -31,6 +31,9 @@ these are required for defining which hosts and guests to run the tests on. - Vagrantfile to use for running vagrant-spec. Unless changed, this should be set as `Vagrantfile.spec`. - VAGRANT_HOST_MEMORY - Set how much memory your host will use (defaults to 2048) +- VAGRANT_SPEC_ARGS + - Specific arguments to pass along to the vagrant-spec gem, such as running specific tests instead of the whole suite + - Example: `--component cli` ## How to run @@ -62,6 +65,24 @@ __Note:__ It is not required that you invoke Vagrant directly in the source repo if you wish to run it else where, be sure to properly set the `VAGRANT_CWD` environment variable to point to the proper test directory inside of the Vagrant source. +### How to run specific tests + +Sometimes when debugging, it's useful to only run a small subset of tests, instead of +waiting for evetything to run. This can be achieved by passing along arugments +using the `VAGRANT_SPEC_ARGS` environment variable: + +For example, here is what you could set to only run cli tests + +```shell +VAGRANT__SPEC_ARGS="--component cli" +``` + +Or with the full command.... + +```shell +VAGRANT_SPEC_ARGS="cli" VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up --provider vmware_desktop +``` + ### About Vagrantfile.spec This Vagrantfile expects the box used to end in a specific "platform", so that it can associate From 0f1f559c5db97c70467caf6030a249fbc1f5aa8a Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Fri, 7 Feb 2020 15:36:15 -0800 Subject: [PATCH 5/6] Update vagrant-spec readme --- test/vagrant-spec/readme.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/vagrant-spec/readme.md b/test/vagrant-spec/readme.md index 1e26bb281..ebce064ac 100644 --- a/test/vagrant-spec/readme.md +++ b/test/vagrant-spec/readme.md @@ -1,10 +1,10 @@ # Running vagrant-spec -The vagrant-spec project is where Vagrant acceptance tests live and are run. +The vagrant-spec project is where Vagrant acceptance tests live. __NOTE:__ You must use a hypervisor that allows for nested virtualization to run these tests. So for the _vagrant_ project, it uses the vagrant vmware plugin as a host. If you want to test this locally, please keep in mind that you will need this hypervisor -to properly run the tests locally. +to properly run the tests. ## Requirements @@ -80,14 +80,14 @@ VAGRANT__SPEC_ARGS="--component cli" Or with the full command.... ```shell -VAGRANT_SPEC_ARGS="cli" VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up --provider vmware_desktop +VAGRANT_SPEC_ARGS="--component cli" VAGRANT_CLOUD_TOKEN=REAL_TOKEN_HERE VAGRANT_HOST_BOXES=hashicorp-vagrant/centos-7.4 VAGRANT_GUEST_BOXES=hashicorp-vagrant/windows-10 VAGRANT_CWD=test/vagrant-spec/ VAGRANT_VAGRANTFILE=Vagrantfile.spec vagrant up --provider vmware_desktop ``` ### About Vagrantfile.spec This Vagrantfile expects the box used to end in a specific "platform", so that it can associate a provision script with the correct plaform. Because some boxes might not end in -their platform (like `hashicorp-vagrant/ubuntu-16.04` versus `hashicorp/bionic64`, +their platform (like `hashicorp-vagrant/ubuntu-16.04` versus `hashicorp/bionic64`), there is a hash defined called `PLATFORM_SCRIPT_MAPPING` that will tell vagrant which platform script to provision with rather than relying on the box ending with the name of the platform. From 65864f86da3ad29b6deb244e4da7ed019f51fa22 Mon Sep 17 00:00:00 2001 From: Brian Cain Date: Fri, 7 Feb 2020 16:15:28 -0800 Subject: [PATCH 6/6] Boost guest memory With 2GB of ram, certain tests were timing out and not able to complete. This commit bumps up that base setting to 5GB to be safe. --- test/vagrant-spec/Vagrantfile.spec | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/vagrant-spec/Vagrantfile.spec b/test/vagrant-spec/Vagrantfile.spec index 8188f1338..f4e52bb87 100644 --- a/test/vagrant-spec/Vagrantfile.spec +++ b/test/vagrant-spec/Vagrantfile.spec @@ -72,7 +72,7 @@ Vagrant.configure(2) do |global_config| config.vm.synced_folder '.', '/vagrant', disable: true config.vm.synced_folder '../../', '/vagrant' config.vm.provider :vmware_desktop do |vmware| - vmware.vmx["memsize"] = ENV.fetch("VAGRANT_HOST_MEMORY", "2048") + vmware.vmx["memsize"] = ENV.fetch("VAGRANT_HOST_MEMORY", "5000") vmware.vmx['vhv.enable'] = 'TRUE' vmware.vmx['vhv.allow'] = 'TRUE' end