On a quick look at the API doc I couldn't find a function for it either.
But you can access it via int config variable
projrecmode using SWS provided functions
SNM_GetIntConfigVar() and
SNM_SetIntConfigVar().
Code:
-- get record mode
recMode = reaper.SNM_GetIntConfigVar("projrecmode", -666)
-- set record mode
reaper.SNM_SetIntConfigVar("projrecmode", 0) -- e.g. auto-punch sel. items
-- hack to trigger record button update
reaper.Main_OnCommand(1007, 0) -- Transport: Play
reaper.Main_OnCommand(1016, 0) -- Transport: Stop
The recording mode symbol of the record button doesn't update immediately when setting it this way it seems, so I've put in start / stop actions to trigger a redraw, maybe there's a more non-hacky way, not sure.
Btw. I used mespotine's
Display altered config-variables script to find the right one, very helpful for stuff like this, props to him.
edit:
I missed that setting the mode can also be done via actions which solves the button update issue.
edit2:
As the 'set record mode' actions report a toggle state alternatively you can use
GetToggleCommandState() to query the recording mode, not requiring SWS.