So here s how I solved this.

First of all, I needed the height of the HTML page to be passed to the pdf generating method, so I added this on the page to be pdf-ed:

<asp:HiddenField ID="hfHeight" runat="server" />

and in the code behind:

protected void Page_Init(object sender, EventArgs e)
{
    if (!IsPostBack)
    {
        string scriptKey = "WidhtHeightForPdf";
        if (!Page.ClientScript.IsClientScriptBlockRegistered(scriptKey))
        {
            StringBuilder sb = new StringBuilder();
            sb.AppendLine("<script>")
                .AppendLine("document.getElementById( " + hfHeight.ClientID + " ).value = document.body.clientHeight;")
                .AppendLine("</script>");

            Page.ClientScript.RegisterStartupScript(typeof(Page), scriptKey, sb.ToString());
        }
    }
}

Now, when I call the pdf generating method I can pass it the height of the HTML. Once I have the height it s all a matter of calculating the width of the pdf "viewport" such that the height fits on the pdf page:

int intHTMLWidth = height.Value * Convert.ToInt32(theDoc.Rect.Width / theDoc.Rect.Height);

And then specify the BrowserWidth parameter either through HtmlOptions of theDoc:

theDoc.HtmlOptions.BrowserWidth = intHTMLWidth;

or when adding the url to theDoc:

int theID = theDoc.AddImageUrl(url, true, intHTMLWidth, true);

EDIT: This solves the question, so I m going to mark it as an answer. Now the next thing to do is create the pdf in protrait or landscape mode based on the width and height of the HTML, so that maximum space is used on the pdf page.

This might be a bit simpler

    /// <summary>
    /// Calculate the height of given html
    /// </summary>
    /// <param name="html"></param>
    /// <returns></returns>
    public int CalculateHeight(string html)
    {
        int id = _Document.AddImageHtml(html);
        int height = (int)(_Document.GetInfoInt(id, "ScrollHeight") * PixelToPointScale);
        _Document.Delete( id );
        return height;
    }

[edit] Well scrollHeight fails with ver8 this works though

    private int AddImageHtml(string html)
    {
        try
        {
            return _Document.AddImageHtml("<div id= pdfx-div-pdf-frame  class= abcpdf-tag-visible  style= abcpdf-tag-visible: true; border: 1px solid red >" + html + "</div>");
        }
        catch (Exception ex)
        {
            throw new Exception(html, ex);
        }
    }

    private double GetElementHeight(int id)
    {
        abcpdf.XRect[] tagRects = _Document.HtmlOptions.GetTagRects(id);
        string[] tagIds = _Document.HtmlOptions.GetTagIDs(id);

        for (int i=0;i<tagRects.Length;i++)
        {
            abcpdf.XRect rect = tagRects[i];
            string tagId = tagIds[i];
            if (string.Equals(tagId, "pdfx-div-pdf-frame", StringComparison.CurrentCultureIgnoreCase))
            {
                return rect.Height;
            }
        }
        return -1;
    }

In case you are using Gecko engine, this engine does not support GetInfoInt so we need write some javascript to get the height. Do a dummy render first to determine height and then set this height to original AbcDoc.

using (var tempDoc = new Doc())
                {
                    tempDoc.HtmlOptions.Engine = EngineType.Gecko;
                    tempDoc.HtmlOptions.Media = MediaType.Print;
                    tempDoc.HtmlOptions.UseScript = true;
                    if (width.HasValue)
                        tempDoc.HtmlOptions.BrowserWidth = width.Value;

                    tempDoc.HtmlOptions.OnLoadScript = " window.onbeforeprint = function () { document.documentElement.abcpdf = Math.max( document.body.scrollHeight, document.body.offsetHeight, document.documentElement.clientHeight, document.documentElement.scrollHeight, document.documentElement.offsetHeight );}";
                    int theTempID = tempDoc.AddImageHtml(htmlData);

                    int height = Convert.ToInt32(tempDoc.HtmlOptions.GetScriptReturn(theTempID));

                    tempDoc.Clear();
                    tempDoc.Dispose();

                    theDoc.MediaBox.Height = height;
                    theDoc.Rect.String = theDoc.MediaBox.String;
                    theDoc.AddImageHtml(htmlData);
                }