How do I know when they are ripe?!?

[harleysilo]

So I've got some cayenne pepper plants loaded with peppers. Over the weeks I've piked some small green ones here and there and nibbled on them. To me they are hot. Make me cough and my mouth water and nose run. Anyways, I thought they were ripe when they turn red. One did which I picked however the skin was tough, not like say a bell pepper when ripe. How do I know when is the proper time to pick?

Same with the Cowhorn, bonnies web site says ready when red, but they taste good when big and green and they are really hot.

Same with Jalapeños, I thought they were supposed to turn red too, but they are great at 3.5" long but still green.

Is it personal preference, or will they be "better" if I wait a little longer?