i tried to configure Rapid STP but could configure two vlan modes(MSTP and Rapid STP) on same core switch.
maybe could not? well.. stp settings are switch-wide, you can not operate in two modes in parallel, however RSTP and MSTP support fallback to native STP.
You do not need to change STP operating mode on any of your switches.
Show full network diagram and full configs.
Why do you have to separate networks? Why not one, in one MSTP region?
If you want fast solution - set bpdufilter on interfaces facing different networks and everything should work, but keep in mind - there is something wrong in your network design.