useragent string and DNN - Value cannot be nul error

[jbarendr]

Since upgrading my DNN installation to v4.9.5, the zoom indexer 5.1.1017 can't access the site. It appears that this is due to user agent not having a version number - I'm not sure if this is a bug in DNN or asp.net.

Is there anyway in Professional version to change the useragent or can you patch it so it has version # ?

For instance, this works OK
Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows)

[David]

I don't have the details handy but I am pretty sure this is a DNN bug. I read the bug report on the DNN developers site a week or so back, but can't immediately find the link to it now. From memory it was fixed in DNN V5.1.

We did some experiments against a DNN site, and found that it was blocking any request from any browser and any spider where the user agent string had certain character string in it. ("Spider" and "Bot" as examples). At least on the site we were looking at it had nothing to do with "missing" version numbers.

If you can't find details on the DNN site, post again and I'll look a bit harder for the link I had to the DNN bug report.

There is no way to change the User Agent in the Pro edition.

[jbarendr]

useragent

I tried using Fiddler and removing 'spider' and that seems to work. However, in Professonal, it won't let me change user string.
I have both 4.9.5 and v5 runnning and both still have this bug. I see DNN posts where this is "fixed". 4.9.4 didn't have problem with Zoom but upgrading to 4.9.5 introduced this bug. v5.1 also has this problem.
You may be correct in bot and spider as:
UserAgent: Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
has version # but has PageLoadException: Value cannot be null. Same error as zoom.
Does Zoom 6 also use ZSEBOT as user agent?

[jbarendr]

more useragent

Did some more digging around and it seems this may be based on browser detection file: http://owenbrady.net/browsercaps/OceanSpiders.browser.xml

Not good at regex stuff - would you be able to tell me how/where to modify this so that Zoom can pass, instead of being blocked? Then I'll try that.

[jbarendr]

Zoom works on DNN

I asked Owen (his email is in OceanSpiders.browser file under App_Browsers how to fix this. He was very helpful and updated the file at his site:
http://owenbrady.net/browsercaps/OceanSpiders.browser.xml

You can either download the file which includes other updates or just add this code to bottom of file:
<!-- Requested by John Barendrecht 10-03-09
-->
<browser id="ZoomSpider" parentID="OceanSpiders">
<sampleHeaders>
<header name="User-Agent" value="ZoomSpider - wrensoft.com [ZSEBOT]" />
</sampleHeaders>
<identification>
<userAgent match="^ZoomSpider" />
</identification>
<capabilities>
<capability name="browser" value="ZoomSpider" />
<capability name="version" value="0.0" />
<capability name="majorversion" value="0" />
<capability name="minorversion" value=".0" />
<capability name="xml" value="true" />
<capability name="crawler" value="true" />
<capability name="w3cdomversion" value="1.0" />
<capability name="tagWriter" value="System.Web.UI.HtmlTextWriter" />
</capabilities>
</browser>
My site at http://oasiscruise.com now works with Zoom search engine. It's running DNN 4.9.5

[Ray]

Glad you've found a solution to your DNN problem.

Does Zoom 6 also use ZSEBOT as user agent?

Yes. More details on Zoom's User-Agent string can be found here:
http://www.wrensoft.com/zoom/support/useragent.html

[David]

We have been asked about this again recently,

There are some details here on the DNN site about it.
http://support.dotnetnuke.com/issue/ViewIssue.aspx?ID=10390&PROJID=2

You can also see errors like "Value can not be null" thrown from DNN for this situation. DNN does a redirect to this invalid page.

The DNN site claims this issue was closed in DNN 5.1.1. but it isn't clear what was fixed if anything.

Here is an example of the problem using Wget with 2 different user agent strings with DNN 4.9.5.

In the bad case a HTTP 302 redirect takes place to send the spider or browser off to an invalid error page.

======== "Bad" user agent ======================
C:\Program Files (x86)\WGet>wget --user-agent="bot" http://5dd.vantegis.com/
--12:02:10-- http://5dd.vantegis.com:80/
=> `index.html.15'
Connecting to 5dd.vantegis.com:80... connected!
HTTP request sent, awaiting response... 302 Found
Location: http://5dd.vantegis.com/Default.aspx?tabid=1673&error=Value+cannot+be+null .%0d%0aParameter+name%3a+String&content=0 [following]

--12:02:10-- http://5dd.vantegis.com:80/Default.aspx?tabid=1673&error=Value+cannot+be+null .
Parameter+name%3A+String&content=0
=> `Default.aspx@tabid=1673&error=Value+cannot+be+nul l.
Parameter+name:+String&content=0'
Connecting to 5dd.vantegis.com:80... connected!
HTTP request sent, awaiting response... 302 Found
Location: http://d44.vantegis.com/?tabid=1673&error=Value+cannot+be+null. [following]

--12:02:11-- http://d44.vantegis.com:80/?tabid=1673&error=Value+cannot+be+null.

=> `@tabid=1673&error=Value+cannot+be+null.'
Connecting to d44.vantegis.com:80...
d44.vantegis.com: Host not found.


======== Different user agent and it is all good ========
C:\Program Files (x86)\WGet>wget --user-agent="LetsBeFriends" http://5dd.vantegis.com/
--12:03:29-- http://5dd.vantegis.com:80/
=> `index.html.15'
Connecting to 5dd.vantegis.com:80... connected!
HTTP request sent, awaiting response... 200 OK
Length: 38,287 [text/html]

0K -> .......... .......... .......... ....... [100%]

12:03:30 (37.39 KB/s) - `index.html.15' saved [38287/38287]
======================================

[David]

More on the topic and a solution for users of the old DNN 4.9.5 package can be found here.

" I copied the app_browser files from a DNN 5.2.x version to my DNN 4.9.5 and it resolved the issue with an indexer with ZoomSearch...."
http://www.dotnetnuke.com/Community/...8/Default.aspx