There is no direct way in a DVE to make a Rounded Corner.
you would need to generate a MASK image that would have the rounded corners and then you would perform a split key of the Video source to the media store with the mask image in it.
Then you would add the DVE but this will use 2 channels as it have to move the video and the mask channels.

------------------------------
Les O'Reilly
Director - Switcher Product Management
------------------------------