Encoding H.264/AAC in MP4 for the PSP:

--profile main
--level 30
--ref 3
--b-pyramid none
--weightp 1

How does this translate into ffmpeg or libav (for the libx264 vcodec)?

ffmpeg:
    -profile:v main -level 3 -x264opts ref=3:b-pyramid=none:weightp=1
libav (avconv):
    -profile:v main -level 3 -refs 3 -b-pyramid none -weightp 1

For 720x480 (NTSC) and 720x576 (PAL), use the MPEG-4 SAR values:
      PAL    NTSC
4:3   12/11  10/11
16:9  16/11  40/33

The mp4 parser _is_ capable of VFR, and decoding capability
frame rate-wise goes up to 60/1.001fps. Just make sure the
timestamps don't overflow over int32. ATOMs in the beginning
treatment preferred, but not required.

HE-AAC is not supported, the decoder will try to decode it as
LC-AAC. Video-only files aren't supported by the player/parser
it seems.

Usable resolutions: 480x272 (and the 4:3 variant of 480xYYY)
                    640x480, 720x480, 720x576
(and most probably a few in between)

Q: Why would I encode in a bigger resolution than the PSP's screen?
A: Because its 4:2:0 YUV -> RGB conversion routines suck.
   This only makes sense in the case the original material is
   as big or bigger than those resolutions, of course, or if
   the same clip is meant to be played on something else
   than the PSP as well.
   
   Of course, if you only care about compression, 480x272/whatever
   should be mechas fine.