Filtering you were trying definitely occurs after the data is returned from RFC call.
Your idea seems worth exploring - try excluding the OU from AOR and putting on the AOR check in an SE37 test first.
If it works, then you can set AOR_CHECK field to X in the data element XML (to save altering the FM code).
If you want to know the inputs to the RFC call, use the RFC Trace feature to see the data passed to the FM, so you can replicate in an SE37 test.
Stephen