shrink.sh script causes build failure on VMware Fusion 7 and Yosemite
#35
Comments
|
Interesting - @lstoll just added the shrink script a few days ago, and this didn't happen to me in testing the latest versions of 10.7-10.10. I suspect it's not an issue with dismissing the GUI popup - Packer's wanting to send the command I think cases where GUI elements / unsaved documents block logout/shutdown is when they're part of timed logout processes initiated at a higher level. If I'm not really surprised this happened, and expect it's the kind of thing that could be intermittent or more likely on different hardware just due to speed differences. Are you able to get this happening repeatedly? Wondering if @lstoll has any suggestions for how the disk shrink step could be modified? With regards to shrinking disks, I used to see a lot of Linux Packer templates do a zeroing out of free space to aid with box compression. I wonder if that's something worth investigating in the shrink script, potentially as an alternative to asking VMware tools to shrink the disk. Currently in the shrink script we don't really deal with any non-zero spare blocks, as far as I can tell, and that could potentially give more mileage at least after the resultant image gets compressed (with potentially a larger uncompressed size however). |
|
Zeroing out free space helps but it is nothing as dramatic as was stated for the shrink script. From my own use for myself using this: It reduces my builds by about a gigabyte (from about 8 gigs to 7, I include a full copy of Xcode.app so reducing size is helpful). I haven't yet merged all the upstream updates to see if I can recreate this issue and if there is a difference in size with both the shrink script and the zeroing things out yet but I could sync things and give it a go by updating the shrink script to zero space too. |
|
I ran the (You might notice that the log looks a little bit different this time. My understanding is that this is because the Virtual Box builder is still running. In the tests described in my original post, I had removed the Virtual Box builder, so the whole I ran this test on the same system. Its specs:
|
- Removed unwanted builders (only `vmware-iso` is left). - Increased disk size. - Bumped darwin version number to match Yosemite. - Increased VM's main memory size. - Bumped VMware hardware compatibility version. - Removed unwanted scripts. - Removed `shrink.sh` script because it causes build failure. See [Issue timsutton#35](timsutton#35).
|
So I got the same issues as dwtj when running on really slow hard drives in my mac mini. When I ran on much faster spinning rust I got the same. Which got me to this packer issue: Which suggested two scripts fields. So I did this (the hack script is all of an echo): And that worked. Versus my normal 10G image I now have a 7.1G image. This is with zeroing free space like so https://github.com/mitchty/osx-vm-templates/blob/myconfigs/scripts/shrink.sh#L12-L14 . Haven't yet looked at the size with just the vmware tool compression. But try out the two scripts fields to see if things work. From my understanding what happens is packer thinks the machine is done executing scripts and then decides to issue a shutdown. By doing a second scripts field it will try to run that script waiting for it to timeout then issue shutdown. I think, either way it works with two scripts fields on vmware. I generated only with packer using -only vmware-iso as well so don't think virtual box has anything to do with the issue. |
|
Nice work! I'll start testing this. Since |
|
Hm. If I define With your template, it actually does both scripts fields? Or am I missing something? |
|
Gah you're right, didn't look through the history far enough. Will try adding it to the end after the shrink script to see if it works or not. This mini is rather slow though so it can take time. As for / vs /dev/rdiskN I recall older versions of diskutil not able to use it. Would have to validate the assumption. |
|
Sorry, I'm away from my computer at the moment. I suspect that the timeout is just the normal packer one, because the shrink does take a bunch of time and will be somewhat IO intensive. I'll be online in a couple of hours, so I'll dig in more and provide some more background on tools shrink vs. zeroing
|
|
Alright so putting the hack script in the same scripts block worked. mitchty@6f30749 Apologies for the escape characters, I was running under mosh (no scroll back) and was tee'ing the output to a log file: But to be honest the difference in using vmware shrink versus just zeroing out things via diskutil isn't all that huge, here is a yosemite box I built last month just using the diskutil zero free space and the box from doing both: ~100 megs isn't a huge win but thats just one data point so not sure if it holds. |
|
If it's just a simple Packer timeout issue when doing the vmware-tools-cli shrink, I'm not surprised I didn't hit it. I've only tested it on a recent iMac i7 with the PCIe SSD. Perhaps there's another template setting we can modify to increase the timeout? |
|
Reading more, it doesn't seem to be the normal timeout so I was wrong there. It also shouldn't be pure computer speed. I'm on a slighter newer 13" rMBP, so I wouldn't expect anything drastic there. I'm running up a test on a Mac mini I have to see if I can reproduce this. As for just zeroing it out, it isn't quite the same. Using vmware tools essentially just zeroes the disk, then re-packs the |
|
Agreed about the diff between vmware tools shrinking and just zeroing. I've often been frustrated by some long-running VMs I have which have slowly ballooned in size, so I should probably try something similar and see how much I can shrink them back down. Wouldn't using |
|
Not really, you can defragment/shrink the vmdk files in place on your hosting os if you want, from an old script I used pre-packer for building arch linux boxes: |
|
Right, I should've said that more clearly. I just meant having to run something on the host as opposed to within the guest during the build process. Ideally there's nothing to have to do after Packer, especially if somoene wants to use Packer's vagrant postprocessor, since this step would need to be done after the build but before the vagrant postprocessor kicks in. |
|
Here's the plugin I was thinking of: https://github.com/shaunduncan/packer-provisioner-host-command |
|
Gotcha I thought you were referring more to the long running vm's. I just use find | xargs after a vm is shutdown to defrag/shrink the vmdk's if they start getting egregious in size. |
|
Is there a solution for this problem ? I ran into that all the time with my MacBook. |
|
I still haven't ever seen Packer actually fail a build because of the VMware shrink process. Not saying it doesn't happen, just that I haven't been able to reproduce it. |
|
Hmm, I think the script is not the problem. It seems there is a second
thread, which is put on hold by a timeout and this timeout is just kicks
in...
So I would like to increase that time, but don't know where..
|
|
But a timeout in Packer? What is taking so long that a timeout gets invoked? |
|
Probably the shrinking. I removed the entry from template.json 2015-12-04 16:11 GMT+01:00 Timothy Sutton notifications@github.com:
Michel Lawaty michel.lawaty@googlemail.com |
|
Host machine: OSX 10.10 Not sure its the exact same issue, but am consistently running into the following error for every OSX packer build:
So far the only workaround I found is to remove the minimize.sh script altogether, but would rather find a better solution f anyone has suggestions? |
guest system: yosemite The host machine is Mac mini (Late 2014), 2,6GHz i5, 8GB DDR3, 1TB HHD |
|
If you remove the I'd like it if we could instead use the local shell provisioner to issue the VMDK shrink using |
|
Packer runs |
|
Oh, ok. I recalled using it to convert DMGs to VMDK, but that must be due to the elevated mechanisms required to do those particular operations. |
|
with |
from unknown reason shirinking operation tend to blow up: timsutton#35
|
Same problem. OSX 1011 guestos, 109 hostos. Edited: Oh, and I use Fusion 8.5 at the moment. Not Fusion 7 as the original poster used. I raised the disk_size to 50200 in template.json, and it crash when shrinking. If I remove shrinking part, it completed with an 21GB box (Xcode included). If I set disk_size to 40960 in template.json it works with shrinking.sh again, but the resulting box has the same size (or even a bit more), 23GB. Is the shrinking thing working at all? Anyway, just wanted to comment that reducing disk_size might help. And that the shrink.sh doesnt shrink my box at all. |
|
Bonus data point: shrinking works fine for me on 10.11 building 10.11 guest. Defaults from template, VMware only. |
|
10.12, building 10.12. Still not working :/ Edited: Oh, and I use Fusion 8.5 at the moment. Not Fusion 7 as the original poster used. ==> vmware-iso: Provisioning with shell script: ../scripts/chef-omnibus.sh
==> vmware-iso: Provisioning with shell script: ../scripts/puppet.sh
==> vmware-iso: Provisioning with shell script: ../scripts/add-network-interface-detection.sh
==> vmware-iso: Provisioning with shell script: ../scripts/autologin.sh
==> vmware-iso: Provisioning with shell script: ../scripts/shrink.sh
vmware-iso: Please disregard any warnings about disk space for the duration of shrink process.
vmware-iso: Progress: 100 [===========>]
vmware-iso:
==> vmware-iso: Stopping virtual machine...
==> vmware-iso: Deleting output directory...
Build 'vmware-iso' errored: Retryable error: Error removing temporary script at /tmp/script_46.sh: dial tcp 192.168.223.191:22: i/o timeout
==> Some builds didn't complete successfully and had errors:
--> vmware-iso: Retryable error: Error removing temporary script at /tmp/script_46.sh: dial tcp 192.168.223.191:22: i/o timeout
==> Builds finished but no artifacts were created.
|
|
Im also having this issue with 10.12.3, building 10.12.3. Fusion 8.6.5: |
I am trying to use to get an OS X VM Running in VMware Fusion. My test system:
template.json)At the end of
packer build template.json, thevmware-isobuilder fails with the following error messages:While the shrink process was running, I noticed that, there was a pop-up box in the guest VM with a warning that there is no disk space left. (The build log obviously says that such warnings should be disregarded). I didn't click "OK" (when I had the chance). I think that it may have been because this pop-up was not closed that the VM could not be shutdown gracefully.
I haven't yet checked whether the user closing this pop-up in time will save the build. However, I have checked that removing
shrink.shfrom the list of scripts in thetemplate.jsondoes allow the build to run to completion. Presumably, removing just the call tovmware-tools-cliwould be a sufficient workaround:The text was updated successfully, but these errors were encountered: