Nice program but ASP.NET control lacks...

[jesper]

I am looking for a new tool / software to add seach to our ASP.NET based web-sites, and came across Zoom Search. Downloaded and tried out the Indexer and was overall very pleased with what I found.

Only puzzling thing in the indexer was that I had to select a text encoding option - isn't it missing an "auto-detect" option? Couldn't it just check the HTTP headers / META tags for this?

Anyway, then it was time to test out the ASP.NET component.
Unfortunately this is a deal breaker in the current state.

Having to place the control outside the main <form> tag or change the form method to "GET" doesn't fly real well with ASP.NET !!
Why not simply also accept POSTs? This isn't any harder to code than GETs.

Also it seems that the index files have to be in the same directory as the search page. I would much prefer to keep these files outside the web-site directory so that they could be updated/uploaded separately from the web-site itself. A simple "IndexFilesLocation" property would do the trick.

If you decide to make these small fixes, I'll be buying a license real quick

[David]

Only puzzling thing in the indexer was that I had to select a text encoding option - isn't it missing an "auto-detect" option? Couldn't it just check the HTTP headers / META tags for this?

It does autodetect the format of the input files. But you need to select the output format that you want & it can't read your mind.

POSTs require a HTML form and submit button for each action. You can't create a HTML stye text link with a POST, but you can with GET. And there are many areas for which we require HTML style links (for example the numeric page navigation links at the bottom of the search results, spelling suggestions, selecting the sort order, etc..).

From the Zoom help file
Some .NET web page generation tools will put a HTML <form> that surrounds the entirety of the document. By default this form submits using POST, whereas this control uses GET variables. It is recommended to move this control outside of the form, but if that is not possible (i.e. the form is on the master page) then try placing the following lines of code within the ASPX code behind load function:

C#
this.Form.Method = "GET";
this.Form.EnableViewState = false;
this.ClientScript.RegisterClientScriptBlock(GetTyp e(), "clr", @"document.
getElementById('__VIEWSTATE').value = '';", true);

Visual Basic

Form.EnableViewState = False
Page.ClientScript.RegisterClientScriptBlock(Me.Get Type(), "clr", "document.

getElementById('__VIEWSTATE').value = ''", True)

Why is this an issue?

Concerning the folders, it makes sense to us to keep all the search related files in a single folder, script and index files. It allows Zoom to update all the file more easily if everything is in one place, especially via FTP.

[jesper]

It does autodetect the format of the input files. But you need to select the output format that you want & it can't read your mind.

I must have misunderstood the UI then - thanks for the clarification.

POSTs require a HTML form and submit button for each action. You can't create a HTML stye text link with a POST, but you can with GET. And there are many areas for which we require HTML style links (for example the numeric page navigation links at the bottom of the search results, spelling suggestions, selecting the sort order, etc..).

Of course - but that's not really my point.
Why not just accept both?

Should simple enough:

If IsPostBack Then
   Query=Request.Form("q")
Else
   Query=Request.QueryString("q")
End If

From the Zoom help file...

That is exactly what I was referring to.

Why is this an issue?

Because if you have any other input fields, or ASP.NET controls relying on the standard POST form, on the same page then you are screwed if the form is changed to GET.
Of course wiping out viewstate like that will also mess up many other ASP.NET controls that you might want on the same page.

Concerning the folders, it makes sense to us to keep all the search related files in a single folder, script and index files. It allows Zoom to update all the file more easily if everything is in one place, especially via FTP.

Sure - but why the same folder as the search page?

If I want to integrate your ASP.NET control with my web-site, the search page will be in the web-site root. That is necessary for relative links and images from the master page to work, and for shorter URLs, etc.
And I really wouldn't want these index files sitting in the root folder - that's just messy.
Preferably they should be completely outside the web-site directory - IMHO.

[Ray]

Of course - but that's not really my point.
Why not just accept both?

The first reason is because there wasn't a need to previously, you're the first person to ask for it. While it would be nice to do and support every thing, we focus on what's in demand.

Second, changing to support POST does not simply mean changing the parsing in of the parameters. It means we have to change the interface as well, the links to the different search result pages, e.g. "<< Previous 1 2 3 ... Next >>", the links to sorting options, the category options, the spelling suggestions.

All of these are text styled links, as commonly expected by most end users, being accustomed to Google. Most people do not expect to click on a form button for these options.

While it is possible to use CSS tricks to disguise form buttons as text links, this is a heavy reliance on CSS that complicates matters for many users who struggle with integrating CSS into their sites and making things work as it is. The support and articles on customizing would be doubly complex.

As usual, there is always a trade off to consider on making the product practical for the majority of our users, and often, our decisions are not as simple as you may think.

Sure - but why the same folder as the search page?

If I want to integrate your ASP.NET control with my web-site, the search page will be in the web-site root. That is necessary for relative links and images from the master page to work, and for shorter URLs, etc.
And I really wouldn't want these index files sitting in the root folder - that's just messy.
Preferably they should be completely outside the web-site directory - IMHO.

It is pretty subjective as to what is "more messy". But I can tell you why we don't do what you say.

Having required files in multiple folders means you'd need to specify multiple paths for the files to be uploaded to. It means multiple sets of files for the user to manage. We have enough users struggling to handle the one set of files, determining the one folder they need to upload them to, and making sure they update the correct set of files after re-indexing (often they've made several copies for testing, and end up uploading to the wrong folder - wondering why their changes aren't reflected in the live search and thinking there's something broken, but in the end it's just because they've uploaded the files to the wrong folder).

To have multiple folders to manage (one folder for the search script, another for the index files), would add unnecessary complexity to this. I personally think that's a messier approach. But some people believe putting all files of a certain type in a folder is "neater" but they often fail to provide an objective reason for it. It's just a pre-established dogmatic idea of what "neatness" is.

If you are worried about the index files being served through the web server, you can quite easily block access to .zdat files (which is default behaviour on an IIS setup and easily done via .htaccess on Apache).

[jesper]

The first reason is because there wasn't a need to previously, you're the first person to ask for it.

Even if no one else has ever asked, I'm pretty sure that any ASP.NET developer who has attempted to use your control in anything but the simplest page would agree with me on this...

Second, changing to support POST does not simply mean changing the parsing in of the parameters. It means we have to change the interface as well...

I don't really see why?
There really shouldn't be any reason to change anything.
All the text based links (Previous, Next, etc.) should still work fine as they are with GET - there is no reason to mess with CSS or javascript for this.
I'm simply suggesting that you support "POST" in addtional to "GET" - not instead of.

As usual, there is always a trade off to consider on making the product practical for the majority of our users

Of course - but I think you'll find that your share of ASP.NET customers will grow if you also support POST...

It is pretty subjective as to what is "more messy".

Yes it is. I have no reason to be objective here.
I'm obviously just telling you what I think - making friendly suggestions as to how your product could be improved for my specific situation.
It is of course entirely up to you to decide if it makes sense for your product and your business to use my subjective suggestions

Of course if you do decide to add an option to specify the index file folder, you could simply let the default behaviour (when the option is not used) be as it is now. That way it still works the same for those "struggling" users who don't need it.

---

On another note - I was playing with your CGI XML/RSS output.
This is actually a lot closer to what I was originally looking for, and I may end up using your product that way instead.

However, this means sending a behind-the-scenes HTTP request to the CGI and then parsing and reformatting the XML output.
It would be even nicer if there was a .NET component (.dll) that could be called in code - returning an object representing the search result - with a similar data layout as the XML/RSS output.
Just another friendly suggestion

[Ray]

I don't really see why?
There really shouldn't be any reason to change anything.
All the text based links (Previous, Next, etc.) should still work fine as they are with GET - there is no reason to mess with CSS or javascript for this.
I'm simply suggesting that you support "POST" in addtional to "GET" - not instead of.

How would the GET requests made by these links not break the VIEWSTATE? That is, when a user clicks on "Next >>", a GET request is sent to the current page. I presume this would break the VIEWSTATE unless a PostBack takes place. Do you know of an alternative approach we haven't thought of?

Of course - but I think you'll find that your share of ASP.NET customers will grow if you also support POST...

Yes, we agree. The current implementation is limited and does not maintain viewstate. It does not use PostBack for all calls to the search script (the aforementioned text links). Also, the current auto-generated search form is inappropriate for such ASP.NET pages as well, and the alternative search form has the same problem.

What we plan to do is update the ASP.NET Server Control to actually make doPostBack calls for all the text links and maintain viewstate. And it will have to pickup the arguments via postback rather than what it does at the moment. I don't think just being able to accept POST parameters as you suggest will be enough, but perhaps I'm missing something.

On another note - I was playing with your CGI XML/RSS output.
This is actually a lot closer to what I was originally looking for, and I may end up using your product that way instead.

However, this means sending a behind-the-scenes HTTP request to the CGI and then parsing and reformatting the XML output.
It would be even nicer if there was a .NET component (.dll) that could be called in code - returning an object representing the search result - with a similar data layout as the XML/RSS output.

You can call the CGI from ASP.NET without a HTTP request as described here:
http://www.wrensoft.com/zoom/support/aspdotnet.html

[jesper]

How would the GET requests made by these links not break the VIEWSTATE?

ASP.NET only uses viewstate for page postbacks (POST).
On GET requests (including requests from text links) the ASP.NET page is loaded fresh - as if this is the first time the page is loaded (no viewstate).

The problem with changing the main <form> tag to "GET" on an ASP.NET page is that this will break any other ASP.NET controls on the page that expect POST and viewstate data.

It is not really a problem having text links pointing back to the same page (like "Previous", "Next", "1", "2", etc.) - resulting in a GET request. Because when a user clicks on one of these links, he is obviously not interacting with any other ASP.NET control on the page.

You could of course make the text links cause a form postback (POST) using some javascript like you describe (which would preserve viewstate for other ASP.NET controls), but I really don't think there is much need for this.
And I, subjectively, don't like a lot of un-needed "messy" javascript
If you do decide to go that way - PLEASE make it optional.

As I see it, all that is really needed to make this work a lot smoother with ASP.NET is just for the control to detect if the request is POST or GET and then get the query string from the form data or query string accordingly, so that the main <form> can stay POST.

You can call the CGI from ASP.NET without a HTTP request as described here...

Very interesting - thanks for the link.

Now does this also work if I put the .cgi and index files in a different folder from the page executing it?
That would solve the other issue I mentioned in my first post.

Also, how does this affect performance? (spawning a new process for every request)


PS: I purchased a pro license last night

[Ray]

ASP.NET only uses viewstate for page postbacks (POST).
On GET requests (including requests from text links) the ASP.NET page is loaded fresh - as if this is the first time the page is loaded (no viewstate).
...
It is not really a problem having text links pointing back to the same page (like "Previous", "Next", "1", "2", etc.) - resulting in a GET request. Because when a user clicks on one of these links, he is obviously not interacting with any other ASP.NET control on the page.

It would seem to me that this would depend on what the other ASP.NET controls store and require from the viewstate. If the page is not expected to be reloaded without a postback, then there is still a risk that those controls will be missing the data they expect, because as you noted, the viewstate is cleared with a GET text link.

And I, subjectively, don't like a lot of un-needed "messy" javascript
If you do decide to go that way - PLEASE make it optional.

Oh, we absolutely despise unnecessary JavaScript. But it seems ironic to request this in the world of ASP.NET. The platform itself uses a great deal of auto-generated Javascript (such as __doPostBack) sometimes without the developer even realizing it. Besides validators, and non-button postbacks, it is also utilized in many of the included controls.

Personally (speaking for myself and not Wrensoft), I'm not very fond of the ASP.NET platform, the bloat of its features (viewstate, etc.), the unnecessary dependencies to achieve very simple things. Someone else here at the office might have a different view.

Very interesting - thanks for the link.

Now does this also work if I put the .cgi and index files in a different folder from the page executing it?
That would solve the other issue I mentioned in my first post.

Also, how does this affect performance? (spawning a new process for every request)

The CGI and the index files need to be in the same folder. But the ASPX page calling the CGI can be in any folder. You just need to change the path in the code that calls the CGI to reflect the location.

The load time of the CGI file (being only 400kb) is negligible compared to the actual search time, so I don't think it would be an issue. Spawning a new process for the CGI is no different to how a normal CGI (i.e. not FastCGI) is executed per request. You can compare how the CGI and ASP.NET executables perform here:
http://www.wrensoft.com/zoom/benchmarks.html#enterprise

[jesper]

It would seem to me that this would depend on what the other ASP.NET controls store and require from the viewstate

Yes of course. There certainly may be some odd case situations where it would be necessary to preserve viewstate also from those links, and I think it would be absolutely wonderful if you implemented this as an OPTION.

My point is that it would take much less modification to your current code to just accept POST in addition to GET - which I think would be sufficient for most ASP.NET scenarios.

One could also argue that when flipping to another page in the search results, you are indeed accessing another "web-page" - and so it it kinda makes sense for other controls to reset.

Oh, we absolutely despise unnecessary JavaScript. But it seems ironic to request this in the world of ASP.NET.

Well - no need to make a bad case worse

Personally (speaking for myself and not Wrensoft), I'm not very fond of the ASP.NET platform

Of course ASP.NET has its faults - just like any platform.
I personally think that the .NET platform as a whole is really great - but then every developer is probably passionate about their own chosen tool set

I understand and respect that ASP.NET is not your primary target audience, but again, I really think you could expand your business in this segment with just a few tweaks as discussed.

The CGI and the index files need to be in the same folder. But the ASPX page calling the CGI can be in any folder. You just need to change the path in the code that calls the CGI to reflect the location.

Perfect - that means that I can keep the .cgi and index files in a folder outside of the web-site.

I'm wondering... is this how your ASP.NET control was made (executing the .cgi behind the scenes)?

Now, I have already written the .NET code to parse the cgi XML/RSS output and turn this into .NET objects - allowing me to do my own output formatting (running at http://www.jhsoft.com/search.aspx).

If you are interested, I'll be happy to wrap this up in a ready to use .NET component (.dll) - with a property setting to specify the cgi/index folder.

I could also help you with the control accepting both GET and POST - all that's really needed is another environment variable to turn off the <form> tags that the .cgi spits out.

The load time of the CGI file (being only 400kb) is negligible compared to the actual search time, so I don't think it would be an issue

Great - thanks.

[Ray]

Of course ASP.NET has its faults - just like any platform.
I personally think that the .NET platform as a whole is really great - but then every developer is probably passionate about their own chosen tool set

I understand and respect that ASP.NET is not your primary target audience, but again, I really think you could expand your business in this segment with just a few tweaks as discussed.

I agree, and do not mean to suggest that ASP.NET does not have its uses. I was just thinking that making post backs optional is kinda like making an elephant go on a diet It's not really going to achieve much. With every configurable option, there are increased complications in terms of support, and usage scenarios. There's always a tradeoff and a decision to determine if it's worth the cost.

Having said that, we might make it optional just for the benefit of backwards compatibility with our existing ASP.NET users.

ASP.NET is an increasing user base for us, and we do take it seriously. The difficult part is that it is still a relatively young platform and we have been getting mixed feedback from the ASP.NET users out there as to what people really want. This is partly due to the fact that ASP.NET is utilized in very different ways by different developers. As the platform matures, and the demand stabilizes, our implementation will adapt to suit accordingly.

As always, the more we hear from ASP.NET users, the more we can help provide what they want. So it would be great if other ASP.NET users would chime in if they find something lacking.

I'm wondering... is this how your ASP.NET control was made (executing the .cgi behind the scenes)?

No, the ASP.NET control is compiled as a native component. It shares code with the CGI (that is, it is coded in C++) and that is why it has comparable performance to the high performance capabilities of the CGI. But it does not execute the CGI separately. It is essentially the CGI with additional code to suit the ASP.NET interface.

Now, I have already written the .NET code to parse the cgi XML/RSS output and turn this into .NET objects - allowing me to do my own output formatting (running at http://www.jhsoft.com/search.aspx).

If you are interested, I'll be happy to wrap this up in a ready to use .NET component (.dll) - with a property setting to specify the cgi/index folder.

Glad to hear you have it working to how it suits you. And feel free to share your wrapper component if you think it will help other users.

[jesper]

Thanks for all you feedback - you do have a very nice product here.

Let me just share my current cgi XML parsing code (VB.NET) - turning the .cgi output into .NET objects for easy output reformatting.
Other ASP.NET users might be able to get ideas from or build on this:

  Function Search(ByVal query As String, _
                  ByVal pageNum As Integer, _
                  Optional ByVal useAnd As Boolean = False, _
                  Optional ByVal category As Integer = -1, _
                  Optional ByVal resultsPerPage as integer = 10) As Result
    Dim qUrl = "http://www.example.com/search.cgi?" & _
               "zoom_sort=0" & _
               "&zoom_xml=1" & _
               "&zoom_query=" & Server.UrlEncode(query) & _
               "&zoom_per_page=" & resultsPerPage & _
               "&zoom_page=" & pageNum & _
               "&zoom_and=" & If(useAnd, "1", "0") & _
               "&zoom_cat%5B%5D=" & category
    Dim wc As New System.Net.WebClient()
    Dim x = wc.DownloadString(qUrl)
    Dim doc = New System.Xml.XmlDocument()
    doc.LoadXml(x)
    Dim root = DirectCast(doc.GetElementsByTagName("rss").Item(0), System.Xml.XmlElement)

    Dim rv As New Result

    rv.TotalResults = Integer.Parse(root.GetElementsByTagName("opensearch:totalResults")(0).InnerText)
    rv.StartIndex = Integer.Parse(root.GetElementsByTagName("opensearch:startIndex")(0).InnerText)

    rv.Items = New System.Collections.Generic.List(Of ResultItem)
    Dim itm As ResultItem
    For Each elem As System.Xml.XmlElement In root.GetElementsByTagName("item")
      itm = New ResultItem
      itm.Number = rv.StartIndex + rv.Items.Count
      itm.Title = elem.GetElementsByTagName("title")(0).InnerText
      itm.Link = elem.GetElementsByTagName("link")(0).InnerText
      itm.Context = ""
      itm.ContextHighlights = New System.Collections.Generic.List(Of ResultItem.Highlight)
      For Each elem2 In DirectCast(elem.GetElementsByTagName("zoom:context")(0), System.Xml.XmlElement).ChildNodes
        If TypeOf elem2 Is System.Xml.XmlText Then
          itm.Context &= DirectCast(elem2, System.Xml.XmlText).InnerText
        ElseIf TypeOf elem2 Is System.Xml.XmlElement AndAlso DirectCast(elem2, System.Xml.XmlElement).Name = "zoom:highlight" Then
          x = DirectCast(elem2, System.Xml.XmlElement).InnerText
          itm.ContextHighlights.Add(New ResultItem.Highlight With {.Start = itm.Context.Length, .Length = x.Length})
          itm.Context &= x
        Else
          Throw New Exception("Context data not recognized")
        End If
      Next
      itm.TermsMatched = Integer.Parse(elem.GetElementsByTagName("zoom:termsMatched")(0).InnerText)
      itm.Score = Integer.Parse(elem.GetElementsByTagName("zoom:score")(0).InnerText)
      rv.Items.Add(itm)
    Next

    Return rv
  End Function

  Public Class Result
    Friend TotalResults As Integer
    Friend StartIndex As Integer
    Friend Items As System.Collections.Generic.List(Of ResultItem)
  End Class

  Public Class ResultItem
    Friend Number As Integer
    Friend Title As String
    Friend Link As String
    Friend Context As String
    Friend ContextHighlights As System.Collections.Generic.List(Of Highlight)
    Friend TermsMatched As Integer
    Friend Score As Integer

    Friend Function ContextToHTML(ByVal beforeHighlight As String, ByVal afterHighlight As String) As String
      Dim rv = ""
      Dim p = 0
      For Each hl In ContextHighlights
        rv &= Server.HtmlEncode(Context.Substring(p, hl.Start - p)) & _
              beforeHighlight & _
              Server.HtmlEncode(Context.Substring(hl.Start, hl.Length)) & _
              afterHighlight
        p = hl.Start + hl.Length
      Next
      rv &= Server.HtmlEncode(Context.Substring(p))
      Return rv
    End Function

    Public Class Highlight
      Friend Start As Integer
      Friend Length As Integer
    End Class