EDIT: Although it sounds like P2: Innocent Sin is sort of a spin-off timeline that spawns into P2: Eternal Punishment which then connects to P3 & P4. From the sounds of everything though I'll be checking P3 out on the PSP, as it seems to be the superior version in terms of "Gameplay" with the exception of some anime cutscenes and the whole "The Answer" thing.
P2: Innocent Sin is an important part of the two 2 games. Eternal Punishment deals directly from the events of that game and even some of the first Persona. I hear it can be enjoyed on it's own, though.
3 and 4 have absolutely nothing to do with the other games except:
- Igor and the Velvet Room is there
- Personas are still taken from normal SMT enemies
- The 1 and 2 characters get mentioned on the TV if you watch on the right day, but not by name and just vague description.
3 is it's own beast with it's own villains and monsters that have nothing to do with the first games, and 4 has nothing at all to do with any of them but:
- The town in the game, Inaba, is named after a P1 or 2 character as an injoke
- You visit the P3 school at one point and meet one social link character from that game and the creepy doctor, along with a few locals from that game with references to STREGA and playing the opening song of P3 FES
You won't be lost at all, I say play 3 since 2 has very different gameplay from both 3 and 4, then play 4 if 2 isn't out yet because it's fantastic.