Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
As Jon says, what you're doing is debatable. Personally, it's the kind of thing I'm willing to do, but Jon is 100% correct, it can be an accident waiting to happen. Regarding such possible accidents, you got to realize the additional risk of connecting any device to the Internet, and treat it with the care it deserves.
Understand such a design exposes your core switch to any and all attacks on its "external" facing interfaces, as they are on the "outside" of the FW. However, if you run those interfaces as pure L2, your switch will be pretty much transparent and, if the switch works correctly, difficult to break into. Perhaps your biggest concern would be some kind of VLAN hopping attack.
If you run the external interfaces as L3, you futher expose your core switch to external attack, but correctly configured interfaces (for security) would mitigate the risk. However, again, as Jon noted, this would be more of an accident waiting to happen, i.e. you "accidentally" start routing traffic between inside and outside without going through the firewall.
From what your diagram shows, why wouldn't you just connect the two ISPs directly to the two FW ports? Are the external VLANs going to host public servers outside the FW? If so, generally you would host them behind a FW on a DMZ VLAN/subnet.
If for some reason, you need switch ports on the outside of the FW, a small L2 or L3 switch would be a good investment from a security perspective. I.e. traffic must transit the FW, there's no physical alternative path.
