This topic comes up every time a new Executable UML forum pops up. :-)
If you look in the help documentation, the metamodel state machine description shows support for both Moore (action on entry) and Mealy (action on transition) support, but as far as I know, the xtUML Editor only supports Moore state machines. It should be noted that Moore and Mealy are 100% convertible to each other.
Harel (nested states) isn’t recommended. Here’s a quote from Marc Balcer, “Executable UML does not employ the full Harel statechart formalism, complete with substates and state history, precisely because such things, in practice, tend to hide abstractions. Modeling is supposed to expose abstractions.”, from the Yahoo! Executable UML group
H.S. Lahman also expressed this view, “Complex state machines usually indicate poor object cohesion (too many responsibilities). [I regard even thinking about using Harel as a similar symptom. B-)]” here.
To sum up the Mellor and Lahman comments, nested states hide the abstracted dynamics of the model, and are only desired, if you’ve improperly abstracted the class model. Exposing more rules of the system in your class models greatly reduces the complexity in the dynamics (states and actions). The most convincing argument is redrawing your Harel state model as a Moore or Mealy state model. ;-)