It is often possible to pass items indirectly without calling attention to yourselves, even when seated next to each other. In the example you give, you might be able to set the food down casually between you.
The concept of kavod ha briyot, human dignity, sometimes allows for halachic leniency in order to spare public embarrassment. Also, a group setting is generally not conducive to the sort of intimacy the harchakot are designed to prevent. So if it is impossible to avoid passing unobtrusively and severe embarrassment would otherwise result, just be sure to avoid touching. Additionally, an effort should be made to have some sort of shinui (change) from the normal. This can be a private signal worked out between the two of you.