The fundamental difference lies in the orientation of the optical components:  

Upright Microscope:

Light source and condenser are located below the stage.

Objectives are positioned above the stage, pointing downwards towards the specimen.

Typically used for examining:

Thinly prepared slides

Small samples

Fixed specimens

Inverted Microscope:

Light source and condenser are positioned above the specimen.

Objectives are located below the stage, pointing upwards towards the specimen.

Designed for:

Examining samples in larger containers  

Easier sample manipulation

Observing living cells in culture