The VBoxManage modifymedium --compact command is currently only implemented for VDI and VHD files. You can't use this command for VMDK files.
Workaround
At the moment the only way to optimize the size of a virtual disk image in VMDK is to clone the image and then use the cloned image in the VM configuration.