Making objects immutable will speed up the runtime, at the cost of a slightly extra compile time hit. There is plenty of work going on to speed up Moose though. The XS work being done on Mouse is planned to be eventually moved over to Moose (gfx is "prototyping" his ideas in Mouse), and Matt Trout is working on making Moose classes "compile" (see his recent blog entry for more info - http://www.shadowcat.co.uk/blog/matt-s-trout/shaving-a-moose/).
Beyond that, I suggest profiling with NYTProf and looking to see where the problem areas are. Some features in Moose cost more then others, for instance, method-modifiers can be expensive since they wrap your sub-calls, and complex type constraints also add some overhead. But without knowing more about your code, there isn't much else I can say.
Also, please note that the Moose community does not watch CPAN-Forums, you will find much more help on either the mailing list or on IRC.