Not entirely sure it s the fastest, but this would likely work:

Rather than placing D half in between the two centers, project C as D along the AB axis. Then check that a) D is between A and B and b) CD is less or equal to half of the rectangle s height.

Re your idea of using the angles... using Pythagorus and or Al Kashi s theorem might actually make sense:

http://en.wikipedia.org/wiki/Law_of_cosines

ABC acute and BAC acute would both be a prerequisite, and given them you place C2 on the rectangle, with the same alpha/beta (see wiki page). From there you also know gamma (pi/2, since on the rectangle) and beta/alpha (pi/2 - beta), which leads to wondering whether the [A,C] or [B,C] distance is lesser or equal to that of [A,C2] or [B,C2] respectively.

The following method is quite simple, but requires finding 1 vector length (you can cache it for multiple checks)

1. Calc vector AB = B - A

2. Calc AB length - it will be the width of the rectangle

3. If AB_length < TOLERANCE (tolerance is a small value, for example, TOLERANCE = 0.00000001) then the rectangle has zero width, therefore the point cannot lie in the rectangle

4. Normalize AB: AB_normalized = AB / AB_length

5. Calculate axis projections

   Calc AB projection:

   AB_proj = dot product(AB_normalized, C - A)

   Calc AB orthogonal projection (denoted "CD" in your picture):

   AB_orthogonal = (-AB.y, AB.x)

   AB_orthogonal_proj = dot product(AB_orthogonal, C - A)

6. If (0 <= AB_proj <= AB_length) and (ABS(AB_orthogonal_proj) <= AB_height / 2), then the point lies in the rectangle, doesn t lie otherwise

The dot product is your trusty friend for problems like this. That and a bit of Pythagoras should give you everything you need to answer two questions:

1. Is the projection of AC onto AB within AB?
2. Is |DC| < height/2?

Work in square distances rather than doing square roots, and don t bother calculating the angle.

The idea with the acute triangle ABC does not work. E.g. it the point C is directly besides the line AB the angle at C will become almost 180°. On the other hand the angle at B might be very small if the rectangle s height is fairly small but C does then lie outside the rectangle.

Nevertheless your other approach is one basic way to implements this.

The point D is somewhere on the line through A and B, thus

D = A + t * (B-A)

(captial letters stand for vectors in space, small letters for numbers). At the same time the connection from D to C is perpendicular to the connection A to B and thus

(C-D) . (B-A) == 0

i.e. the dot product of the two difference vectors is zero. Putting both together yields

(C-A-t*(B-A)) . (B-A) = (C-A) . (B-A) - t * (B-A) . (B-A) == 0

or when solved for t:

t = (C-A).(B-A) / (B-A).(B-A)

(or in other words the relative length of the projection of the vector AC onto the line AB).

The point D is within the rectangle if 0 <= t <= 1, and thus this is your first condition.

Afterwards you can calulate the distance of C to D (just insert t into the very first eqn to get D) and compare this to h/2, i.e. the last condition then is

(C-D).(C-D) <= (h/2)^2

The square is the intersection of 4 half-spaces. Each half-space is derived from one side of the rectangle using the two-point line formula. Depending on the particulars of your problem, perhaps you can check side-by-side, possibly trivially rejecting at each step.

Is this faster than projection? I guess that depends on the probability that you ll trivially reject after your first step!