When you are using big program like Firefox — yes, it's big, some of memory usages may go into swap. However, if your harddisk's performance is like mine, it's totally in pain. One word only can describe “slow.”

There is a way to move the content in swap back to main memory, run the following as root:
swapoff -a
swapon -a

Doesn't look genius to me, but I don't know if there is any way else. If it is, please leave a comment.

You may also want to change the system setting, so reduce the chance of moving back. On my Fedora 10, the default vm.swappiness is 60. You can change it on-the-fly as root:
sysctl vm.swappiness=10
And edit the /etc/sysctl.conf as root:
vm.swappiness = 10

If you don't want to use swap anymore, after you swapoff, then to remove the swap partition by removing from /etc/fstab and fdisk'd.