As you add rules to a single mail filter, the compose window grows vertically to accommodate the increasing list of rules.
No vertical scroll bar appears in this window ever, so if you add enough rules you block access to both the title bar at the top and the control buttons below.
At that point, you are stuck! You must either F11 (if you haven't done so already) to maximize the browser or delete some rules to get the control buttons back into view. And then you have to create a new filter to accommodate the additional rules you were trying to add in the first instance.
I just opened bug 11911 for this here.
Would be nice if this window gets a vertical scroll bar after it reaches a certain size.
All the best,