Consider using an open source 3d engine like Ogre3D or Irrlicht. This way you can focus on making a game: content, gameplay.

Both engines define an abstraction layer that can map to Direct3D or OpenGL backends.

EDIT: at first I didn t notice you mentioned Java, anyway both engines support Java bindings.

If you decide to learn the low level graphics programming instead of using a higher level 3d engine like Ogre3D or Irrlicht, I recommend starting with OpenGL ES 2.X, which is a subset of OpenGL.

OpenGL ES 2.X is a simplified subset of OpenGL for embedded systems with programmable hardware such as phones. For example, in OpenGL there may be multiple ways to accomplish tasks, but in Open GL ES 2.X, there s only one way. Because there s less choice in OpenGL ES 2.X, there s so much less to learn to master OpenGL ES 2.X. The OpenGL ES 2.0 Programming Guide is 480 pages, and the OpenGL Programming Guide, Seventh Edition is 936 pages.

Furthermore, using OpenGL ES 2.X will give you a wider hardware audience for your game. Because it is a subset of OpenGL, code written for OpenGL ES 2.X will work on a full OpenGL system with little or no modification. Your game s graphics can work on a range of target machines from desktops to smart phones.

You could choose an open source 3D engine like Ogre3D.

But if you really want to write the graphics engine on your own (quite interesting), start by learning linear algebra. Not the quite advanced stuff, but you should be very comfortable with expressing yourself in translation vectors, rotation matrices and the like.

If you already familiar with those concepts, choose either Direct3D or OpenGL - this depends more on the available platform. If you like to code in a Mircosoft Environment, choose Direct3D, everything else, choose OpenGL.

The choice of additional libraries depends on the programming language. Better alternatives than GLUT are probably SDL (C) or Qt(C++).

Well, if you ve never done 3D graphics programming before, go for simplicity and learn-as-you-go. There used to be a set of really nice tutorials by NeHe (Neon Helium) for OpenGL, which was the route I took.

I found OpenGL easier to start with as learning curve is not so steep. You can get quick results which helps a lot with initial frustration/motivation. Tons of help etc. Words of warning though - 3D programming is inherently complex and you ll spend more energy on building graphics technology than game itself. Consider adopting Ogre3D if you really want to see the game finished :o) (disclaimer: I know there are also other engines but Ogre is stable, proven and comes with good documentation)

If you want to stay platform-independent DirectX won t do as it is restricted to Microsoft Windows Platforms