rdesktop-vrdp crashes with memory corruption - regression from 3.2.12-68302 -> fixed as of 14 Jan 2011

[Paul Kronenwetter]

rdesktop-vrdp -r usb {host}

Autoselected keyboard map en-us
WARNING: Remote desktop changed from 800x600 to 1020x550.
*** glibc detected *** rdesktop-vrdp: free(): invalid next size (fast): 0x000000000236a930 ***
*** glibc detected *** rdesktop-vrdp: malloc(): memory corruption: 0x000000000236a950 ***

From here the window doesn't update and sometimes causes the rdesktop client (Fedora 14) to freeze / crash.

Problematic rdesktop-vrdp is from VirtualBox-4.0-4.0.0_69151_fedora14-1.x86_64.rpm. Previous rdesktop-vrdp from VirtualBox-3.2-3.2.12_68302_fedora14-1.x86_64.rpm works perfectly.

[Frank Mehnert]

Any easy way to reproduce this crash?

[Paul Kronenwetter]

The glibc memory corruption occurs almost immediately, but only when redirecting USB devices. I have only seen the system freeze symptom a few times and not recently.

[Paul Kronenwetter]

Ok, there may be another factor involved. Either it's due to my USB scanner (from NeatWorks) or the fact that I'm using IPv6 at home and not from work... Let me investigate further and report back.

[Paul Kronenwetter]

Backtrace from failed session.

[Paul Kronenwetter]

I've tested a few scenarios and it's not IPv6. I've attached a backtrace though I don't get one every time. The command that generated it was: $ rdesktop-vrdp -r usb -z -P -a 16 host:3407
Though the options don't seem to matter much but the USB device connection / disconnection is important.
Let me stop rambling...

Here's a sequence of steps with outcomes:

Start the VM (WinXP SP3) and attach rdesktop-vrdp -r usb host:3407 while the VBox splash screen was still visible. XP saw the USB devices.
Unplug the scanner and rdesktop crashes.
Re-connected rdesktop, crashed almost immediately.
Re-plugged in the scanner and reconnected rdesktop, it was fine but no USB devices appeared.
Rebooted Win XP, still no USB devices.
Completely shut down VM, started VM and reconnected with scanner attached, devices are seen.
Detached rdesktop, unplugged scanner, re-connected rdesktop. Crash again.
Detached rdesktop, re-plug scanner, connect with rdesktop-vrdp from 3.2_68302.
Do all of the above stuff, never crashes, all works as expected.
Detached rdesktop, re-connect with 4.0's rdesktop-vrdp and with scanner connected, seems happy.

Hopefully this is something you can use. I can test with a debug version (64-bit) or try other combinations.

[Paul Kronenwetter]

VBox log file from last comment.

[Michael Thayer]

If you can reproduce this using

​http://www.alldomusa.eu.org/download/testcase/rdesktop-vrdp

which contains debug symbols, could you try running it in valgrind to see what output is produced? Thanks.

[Michael Thayer]

Did the run under Valgrind also abort with the error you reported initially? I am slightly confused, as I can't see any trace of it in the output.

[Michael Thayer]

And could you also provide the output of "VBoxManage list usbhost" with and without the scanner plugged in, in a situation in which you can reproduce the crash? Thanks again.

[Paul Kronenwetter]

I honestly don't remember... Sorry. I did get it to crash this time, so I'll replace the log file. Looks much more interesting. I'll try to get the list usbhost, but it doesn't give me a lot of time before it bombs...

[Paul Kronenwetter]

Valgrind output with debug-enabled binary - updated 7-Jan-2011

[Paul Kronenwetter]

Unless I'm mistaken, list usbhost only shows the USB devices actually attached to the host. I have two of them, but they're attached to a different VM and they're not the scanner involved. That's why I'm using RDP with USB extensions. :)

[Michael Thayer]

You are right of course - is it possible to install VirtualBox on the system you are attaching the scanner to, in order to do that test?

[Michael Thayer]

Although hopefully the Valgrind output will be enough to fix this - this time it does look like what I need.

[Paul Kronenwetter]

Um yes, though it'll be *much* slower. I'll see what I can do and get back to you.

[Michael Thayer]

Don't worry, I think I have fixed this locally. More to follow.

[Michael Thayer]

I have uploaded a version of VirtualBox containing a hopefully fixed copy of rdesktop-vrdp. This is a test build from the stable branch of VirtualBox (disclaimer here). It is a distribution-agnostic build with a shell script installer. To unpack it without installing, do

$ ./VirtualBox-*.run --keep --noexec

which will create a subdirectory called "install" in the current directory with the installer files, including a tar.bz2 containing the actual executables. The URL for the test build is

​http://www.alldomusa.eu.org/download/testcase/VirtualBox-2011-01-13-16-01-33-lin64-rel-4.0.1-r69417.run

[Paul Kronenwetter]

My testing shows that this looks like it fixed it. I've pulled and re-inserted the scanner a few times, no crashes. Thanks!

[Michael Thayer]

Thanks for testing.