You re looking for a non-trivial solution v to A*v=v with v=[x;y;z] and...

A =
   0.70710678118655                  0   0.70710678118655
  -0.50000000000000   0.70710678118655   0.50000000000000
  -0.50000000000000  -0.70710678118655   0.50000000000000

You can transform this into (A-I)v=0 where I is the 3x3 identity matrix. What you have to do to find a nontrivial solution is checking the null space of A-I:

>> null(A-eye(3))

ans =

   0.67859834454585
  -0.67859834454585
   0.28108463771482

So, you have a onedimensional nullspace. Otherwise you d see more than one column. Every linear combination of the columns is a point in this null space that A-I maps to the null vector. So, every multiple of this vector is a solution to your problem.

Actually, your matrix A is a rotation matrix of the first kind because det(A)=1 and A *A=identity. So it has an eigenvalue of 1 with the rotation axis as corresponding eigenvector. The vector I computed above is the normalized rotation axis.

Note: For this I replaced your 0.7071 with sqrt(0.5). If rounding errors are a concern but you know in advance that there has to be a nontrivial solution the best bet is to do a singular value decomposition of A-I and pick the right most right singular vector:

>> [u,s,v] = svd(A-eye(3));
>> v(:,end)

ans =

   0.67859834454585
  -0.67859834454585
   0.28108463771482

This way you can calculate a vector v that minimizes |A*v-v| under the constraint that |v|=1 where |.| is the Euclidean norm.

I don t think you need to use the solve function as your equations is a system of linear equations.

Recast as a matrix equation:

Ax = B

In your case:

    | -0.2929   0.0      0.7071  |  | x |     | 0 |
    | -0.5     -0.2929   0.5     |  | y |  =  | 0 |
    | -0.5     -0.7071  -0.5     |  | z |     | 0 |

Use the built-in functionally of MATLAB to solve it. See e.g. MATLAB: Solution of Linear Systems of Equations.

The core of MATLAB is to solve this kind of equation.

Using FreeMat (an open-source MATLAB-like environment with a GPL license; direct download URL for Windows installer):

   A = [ -0.2929 0.0 0.7071; -0.5 -0.2929 0.5; -0.5 -0.7071 -0.5 ]

   B = [0.0; 0.0; 0.0]

   AB

   ans =
    0
    0
    0

So the solution is: x = 0, y = 0, z = 0

The solution can also be derived by hand. Starting from the last two equations:

    -0.5*x + 0.7071*y +    0.5*z = y
    -0.5*x - 0.7071*y +    0.5*z = z

    0.2929*y =  -0.5*x + 0.5*z
    0.7071*y =  -0.5*x + 0.5*z

    0.2929*y = 0.7071*y

Thus y = 0.0 and:

    0.7071*y =  -0.5*x + 0.5*z

    0 =  -0.5*x + 0.5*z

    0 =  -0.5*x + 0.5*z

    0.5*x = 0.5*z

    x = z

Inserting in the first equation:

    0.7071*x + 0.7071*z = x 

    0.7071*x + 0.7071*x = x 

    1.4142*x = x 

Thus x = 0.0. And as x = z, then z = 0.0.

x = 0, y = 0, z = 0 is the correct solution. This problem can be easily worked by hand.

A = [ 0.7071 0 0.7071 ;
      -0.5 0.7071 0.5 ;
    -0.5 -0.7071 0.5 ];
B = [1 ; 1 ; 1];

AA = A-diag(B)

      -0.2929            0       0.7071
         -0.5      -0.2929          0.5
         -0.5      -0.7071         -0.5

BB = B-B

     0
     0
     0

AABB

     0
     0
     0

Friends use MATLAB command rref(A) for Ax=B.... It will give a upper triangular Matrix. Then u can calculate non-trivial solutions easily... But keep in mind that if rref(A)=I (as in your case) then non-trivial solutions do not exist. BEST OF LUCK