What kind of encryption are you looking for? Are you looking for raw RSA encryption, or any specific message format?
Out of the box, BizTalk only supports S/MIME encryption using the SMIME encoder/decoder component; it might be useful depending on exactly your format is.
As for how to create a custom pipeline component from scratch to do it, I recommend first starting with the Pipeline Component Wizard. It will take care of most of the boilerplate code.
I do have a sample on writing custom encryption pipeline components, though my specific sample uses symmetric encryption and not RSA (but should give you a clue as to how to implement this). The code for these components can be found here.