I discovered the following doing the opposite to what you are trying to achieve, i.e. changing a service library to a console application..

some of the settings in the csproj files cannot be edited from the settings screen from within VS to convert an class library to a WCF Service Library you need to add the following to your project file

Add the following to the first PropertyGroup [these are the guids for a C# WCF Project]

<ProjectTypeGuids>{3D9AD99F-2412-4246-B90B-4EAA41C64699};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

See here for further information on ProjectTypeGuids

You may also need to add the following line immediately below:

<StartArguments>/client:"WcfTestClient.exe"</StartArguments>

But ultimately it s the PropertyTypeGuids that you need to manually insert to get VS to recognise the project as a WCF Service Library Project.

This is what I had to do to convert my class library to WCF REST application.

1) Modify the .csproj file and add the below two lines to the first PropertyGroup element in .csproj file.

<ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>
<UseIISExpress>false</UseIISExpress>

2) Add the following line to below <Import Project="$(MSBuildBinPath)Microsoft.CSharp.targets" /> to import Microsoft.WebApplication.targets file

<Import Project="$(MSBuildExtensionsPath32)MicrosoftVisualStudiov10.0WebApplicationsMicrosoft.WebApplication.targets" />

3) Add the following code to the end of the file before the </Project> tag.

<ProjectExtensions>
<VisualStudio>
  <FlavorProperties GUID="{349c5851-65df-11da-9384-00065b846f21}">
    <WebProjectProperties>
      <UseIIS>False</UseIIS>
      <AutoAssignPort>True</AutoAssignPort>
      <DevelopmentServerPort>50178</DevelopmentServerPort>
      <DevelopmentServerVPath>/</DevelopmentServerVPath>
      <IISUrl>
      </IISUrl>
      <NTLMAuthentication>False</NTLMAuthentication>
      <UseCustomServer>False</UseCustomServer>
      <CustomServerUrl>
      </CustomServerUrl>
      <SaveServerSettingsInUserFile>False</SaveServerSettingsInUserFile>
    </WebProjectProperties>
  </FlavorProperties>
</VisualStudio>

4) Save the .csproj file and Reload the project.

5) Add a Web.Config file to the project and add the below bare minimal code. You can add more later per your requirement.

    <?xml version="1.0"?>
<configuration>

  <system.web>
    <compilation debug="true" targetFramework="4.0" />
  </system.web>

  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true">
      <add name="UrlRoutingModule" type="System.Web.Routing.UrlRoutingModule, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
    </modules>
  </system.webServer>

  <system.serviceModel>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
    <standardEndpoints>
      <webHttpEndpoint>
        <!-- 
            Configure the WCF REST service base address via the global.asax.cs file and the default endpoint 
            via the attributes on the <standardEndpoint> element below
        -->
        <standardEndpoint name="" helpEnabled="true" automaticFormatSelectionEnabled="true"/>
      </webHttpEndpoint>
    </standardEndpoints>
  </system.serviceModel>

</configuration>

6) Add a Global.asax file. Below is a sample file.

    public class Global : HttpApplication
{
    void Application_Start(object sender, EventArgs e)
    {
        RegisterRoutes();
    }

    private void RegisterRoutes()
    {
        // Edit the base address of Service1 by replacing the "Service1" string below
        RouteTable.Routes.Add(new ServiceRoute("YourService", new WebServiceHostFactory(), typeof(YourServiceClass)));
    }
}

7) Finally in the project s properties, under Build tab, if the output path is set to binDebug modify it to bin.