You need to ask the vendor how to configure their hardware. On the Cisco side of things it should be a standard SIP trunk toward the application server running the Brooktrout software and enabling T.38 Fax Relay on your PSTN gateway. If the vendor has some non-standard SIP configuration that they want it's pretty standard for them to document how they want CUCM configured. Even Microsoft does this with Exchange UM - including screenshots - so expecting them to have clear documentation shouldn't be asking much.
! For standard 14.4kbps V.30 faxing
voice service voip
fax protocol t38 version 0 ls-redundancy 2 hs-redundancy 2 fallback pass-through g711ulaw
modem relay nse codec g711ulaw gw-controlled
! If you want to enable 33.6kbps V.34 faxing
voice service voip
fax protocol t38 version 3 ls-redundancy 2 hs-redundancy 2 fallback pass-through g711ulaw
modem relay nse codec g711ulaw gw-controlled
no fax-relay sg3-to-g3
Ensure that the ls-redundancy and hs-redundancy parameters match what you configure on the Brooktrout side of things. Set them to zero (0) if it doesn't support packet redundancy.
It is also useful to clear the NSF field to avoid any proprietary vendor features that don't comply with the T.30/T.34 standard and could cause T.38 fax relay to fail.
! Dial-peer facing CUCM from the PSTN gateway
dial-peer voice <tag> voip
destination-pattern +.*
session protocol sipv2
session server-group <tag>
session transport udp
incoming uri via <tag>
destination dpg <tag>
voice-class codec <tag>
voice-class sip profiles <tag>
voice-class sip options-keepalive profile <tag>
dtmf-relay rtp-nte sip-kpml
ip qos dscp cs3 signaling
no vad
fax nsf 000000
This answer assumes current versions of both IOS and CUCM.
