First of all, you'll need a bat of your own. Sure, you can always borrow other people's, but it's best to have your own personal ping-pong paddle. I'll talk more about how to choose your first table tennis racket later, but for now I'm simply going to describe what a table tennis racket actually is, without getting too bogged down in all the rules concerning rackets just yet (and there are quite a few!).
Then, either sandwich rubber or ordinary pimpled rubber is glued onto the sides of the blade that will be used to hit the ball. These rubbers are colored red or black, and the color on one side must be different from the other side (i.e. one red side, one black side). If one side is left without rubber, you must not hit the ball with this side, and it must be colored red if the rubber on the other side is black, or vice versa.
An ordinary pimpled rubber is made up of a single layer of non-cellular rubber, with pimples evenly spread out over its surface.
A sandwich rubber is made up of a layer of cellular rubber, to which another lay of pimpled rubber is glued on top. The cellular rubber (or sponge) is glued to the blade, and the layer of pimpled rubber is used to hit the ball. The pimples may face inwards or outwards. If the pimples are facing outwards, this is called a pimples-out (or pips-out) sandwich rubber. If the pimples are glued to the sponge, this is called a pimples-in sandwich rubber, reverse rubber, or smooth rubber.
The most common rubber in use today is smooth rubber, which generally gives the most spin and speed when hitting the ball. However, pimples-out sandwich rubber is still used by certain players due to its good speed and better control for hitting against spin. Ordinary pimpled rubber is rarer due to the lack of spin and speed it can produce, but is an option for some players who prefer its greater control (when ordinary pimpled rubber is used on both sides of the blade, this is called a hardbat).
Ping-pong balls can be purchased from many sports stores, although most clubs will purchase them from table tennis dealers. Balls of 40mm diameter are now used, so be careful that you aren't playing with any old 38mm balls that you might have had lying around for years!
The balls are usually made of celluloid, and are white or orange when used in competitions.
Most manufacturers grade their balls according to a 3 star system. 0 star and 1 star balls are usually used for training purposes, since they are cheap and quite acceptable for this type of play. They are the lowest quality balls, but the 0 star balls from manufacturers such as Stiga, Butterfly or Double Happiness are actually surprisingly good these days.
2 star balls are supposed to be of better quality than the 0 and 1 star balls, but still not considered good enough for serious competition. In reality these balls are rarely seen or used - I can't remember ever seeing more than a couple of 2 star balls!
3 star balls are the competition standard balls, and are the best quality. Occasionally you will get a not quite round 3 start ball, but it's rare. They are almost always a good roundness and balance. They are quite a bit more expensive than the 0 or 1 star balls though, and they don't seem to last any longer either!
Some manufacturers such as Stiga and Nittaku are now making what is called '3 star premium' balls. These are supposed to be of the highest quality possible. Whether this is really true or just another bit of marketing hype is open to debate - I know that I can't tell the difference between a 3 star and a 3 star premium ball.
Don't bother starting out with 3 star balls or 'premium' balls - they are too expensive and not really worth it for beginners. Simply buy some 0 or 1 star balls from a reputable manufacturer such as Butterfly or Stiga and these will do perfectly well. You also won't feel like crying if you accidently step on one!