According to what I've read on the web, this should work:
(hwnd is a top-level window with a GWL_STYLE of WS_THICKFRAME)
Code:
NSWindow* window = (NSWindow*)hwnd;
[window setAlphaValue:opacity];
[window setOpaque:NO];
However, instead of affecting the entire window, only some parts (probably some child windows) are changed, and instead of becoming transparent (i.e. showing other windows underneath) these parts fade to opaque white.