You can use some fake knives to spring out of the board (springload them inside the board), then cover the hole you've got them coming out of with balloons.  Put a small pin on the handle edge of the knife that pops the balloon when the knife emerges from the board.  Some pins will miss balloons and that will simply look like the knife thrower went a bit to the side.

 

Knife thrower has his own knife that he palms as he throws it.  When he reaches for the next knife, he simply unpalms the existing knife

 

Fake retractible knife props are about $2 online. Here's one source  http://www.dealextreme.com/details.dx/sku.8579