I think I can now 100% confirm that VMwareHostOpen.exe on 64-bit systems CANNOT open files with short path names in them! Here is how you can verify this:
[1] Make sure you have a shared folder with the host, e.g., turn on mirroring for Downloads directory.
[2] Using Windows Explorer, create a new text file, named 12345678.txt in Z:\Downloads directory.
[3] Open the new file in Notepad and enter some test text, e.g., TEST. Save and close.
[4] Open a command prompt. Type the following to confirm that there IS NO 8dot3 name generated:
C:\Users\Han Sooloo\Downloads>dir /x z:\Downloads\1*
Volume in drive Z is Shared Folders
Volume Serial Number is 0000-0064
Directory of z:\Downloads
03/31/2013 02:27 PM 4 12345678.txt
1 File(s) 4 bytes
0 Dir(s) 13,058,822,144 bytes free
[5] Use VMwareHostOpen to open the file on the host:
C:\Users\Han Sooloo\Downloads>"c:\Program Files\VMware\VMware Tools\VMwareHostOp
en.exe" --file 12345678.txt
[6] Not that TextEdit (or your registered editor) will open the file and you will see your test text, e.g., TEST.
[7] Now, go to Windows Explorer, and make the file name long by adding a "9" to the end, i.e., 12345678.txt ==> 123456789.txt.
[8] Confirm an 8dot3 equivalent short file name is generated for the renamed file.
C:\Users\Han Sooloo\Downloads>dir /x z:\Downloads\1*
Volume in drive Z is Shared Folders
Volume Serial Number is 0000-0064
Directory of z:\Downloads
03/31/2013 02:27 PM 4 1234~749.TXT 123456789.txt
1 File(s) 4 bytes
0 Dir(s) 13,058,109,440 bytes free
[9] Use VMwareHostOpen again to try to open the file on the host:
C:\Users\Han Sooloo\Downloads>"c:\Program Files\VMware\VMware Tools\VMwareHostOp
en.exe" --file 1234~749.txt
[10] Notice that *NOTHING* happens, and the file is NOT opened in the host's text editor.