This is a standard 3D-geometry problem. A satellite is visible, if it is on the other side of the plane that touches the earth in the location of observation (tangential plane). This plane is best described by its normal vector, which is just the vector from the earth's center to the point of observation. To see, on which side of the plane the satellite is, amounts to calculate a scalar product of cartesian coordinates (which have to be calculated first). A common mistake is to take the vector from the center of the earth to a satellite instead of the vector from the location of observation to it.