So:
In Notes 5.x, you are able to control the expansion of public groups by adding a $ExpandGroups item to the message before pressing F9. To add the $ExpandGroups item, you can modify the design of the memo form (or use an alternate mail form that includes this item) or add the item via an agent or button. If the item exists when F9 is pressed, the following values for the item will be interpreted by the mailer to alter the default group expansion behavior:
"0" - Do not expand private or public groups
"1" - expand private groups only (same as default behavior)
"2" - expand public groups only
"3" - expand private and public groups
This functionality is also available in Notes 4.6x, but the $ExpandGroups item must be of TYPE_TEXT. In Notes 5.x, the item can be either TYPE_TEXT or TYPE_TEXT_LIST.