Issues in Statistics report

[Tapio]

I have many problems using the statistics generated by 'report.php' (Zoom v8, build 1017). Basically it works, but the PHP (7.3.33) is throwing a lot of Notices and Warnings, depending on selected time span, end date format and even on reporting weekday.

1) There is a difference in Warnings if I type the time span end date "now" or in a date format (e.g. 2022-06-14). Should be no difference.
2) The monthly result gives Warnings, if the time span does not cover a full calender month.
3) The weekly result gives Warnings, if the time span does not cover a full calender week.
4) The weekly result gives Notice if I have a look at the report on Monday.

The Notices and Warnings appear in these places in report.php:

a) Searches per Month (time span not full month):
Notice: Undefined variable: assoc_arr - - \report.php on line 954
Warning: Invalid argument supplied for foreach() - - \report.php on line 954
Notice: Undefined variable: output_array - - \report.php on line 977
Warning: array_reverse() expects parameter 1 to be array, null given - - \report.php
Warning: Invalid argument supplied for foreach() - - \report.php on line 1155
Warning: Invalid argument supplied for foreach() - - \report.php on line 1181
Warning: count(): Parameter must be an array or an object that implements Countable in - - \report.php on line 981
Warning: Division by zero - - \report.php on line 981

b) Searches per Week (time span not full week):
Notice: Undefined variable: assoc_arr - - \report.php on line 865
Warning: Invalid argument supplied for foreach() - - \report.php on line 865
Notice: Undefined variable: output_array - - \report.php on line 886
Warning: array_reverse() expects parameter 1 to be array, null given - - \report.php on line 886
Warning: Invalid argument supplied for foreach() - - \report.php on line 1155
Warning: Invalid argument supplied for foreach() - - \report.php on line 1181
Warning: count(): Parameter must be an array or an object that implements Countable in - - \report.php on line 890
Warning: Division by zero - - \report.php on line 890

=========

Then a feature request:
I'd like to make the report public. But in Raw Data -report I prefer to hide the IP-addresses for privacy. So far I have not found a way to do /hack it.

[Simon (PassMark)]

We have made some changes to the report.php file that should fix the warnings/notices.
You can download updated file here

Also added $RAW_NOIP which you can change in the Customizable Settings section to hide IP addresses.

Let us know if you are still getting any warnings/notices.

[Tapio]

Thanks, it is better now. But there are still cases where I do get Notices /Warning from PHP:

1) In some cases (e.g. if there is no search data on ending date, period is short or something else?):
- @Week Report -> Notice: Undefined index: 2022, Wk 24 in - - \report.php on line 844
- @Week Report -> Notice: Undefined variable: assoc_arr in - - \report.php on line 844
- @Week Report -> Notice: Undefined offset: 0 in - - \report.php on line 875

2) If the end date is before start date in the time period:
- (twice -> Warning: date_format() expects parameter 1 to be DateTimeInterface, int given in - - \report.php on line 194

[Simon (PassMark)]

Made further changes to the report.php file to address the above notices/warnings.
Download using the same link here

[Tapio]

Getting better again. But now I found one type of report, which I had totally forgotten to test, and that seem to have issues as well:

"Search phrases returning 0 results" ->
Phrase - Count - Percent
All other phrases 0 NAN%
Total 0 NAN%

Happens only, when the logfile is missing any searches giving zero results:
- Warning: arsort() expects parameter 1 to be array, null given in - - \report.php on line 700
- Warning: Invalid argument supplied for foreach() in - - \report.php on line 705
- Warning: Division by zero in - - \report.php on line 722
- Warning: Division by zero in - - \report.php on line 725

After these fixes I believe the report.php is working OK.

=============

Then I came across an interesting issue in the logfile:
On every day there are few searches, either phrase "2022" or "2023" (without quotes). In our case this search does not make any sense, as these phrases are found only on the page number of the book. Anyway, according to Raw Data, these searches come from three different IP's, which all refer to same source: "msnbot-[IP].search.msn.com".

So far not very mysterious from a bot. But in the Raw data there are 1-2 strange things:
- And/Or is switched to AND from default OR (for a single word).
- Results per page is 10. But this choice does not exist in my "search.php", as I have modified the dropdown-list to include only 25 or 100 results per page.

Can this bot use some other motor and marking our logfile, although our logfile name is not default, but rather peculiar?

[Simon (PassMark)]

Above warnings should be fixed now, download here

Not too sure, maybe there is a link on some page that it can access that has url parameters set to use AND and 10 results on one page. E.g. I can set results per page to 22 (zoom_per_page=22) and AND (zoom_and=1):
https://www.zoomsearchengine.com/sea...m_cat%5B%5D=-1

Maybe check your server logs to see what pages the bots are trying to access.

[Tapio]

The report-file is now OK.

And thanks for advice. But is there any way to block external url's to fill the searchlog with garbage? I'd like only queries from my own site /searchform (search.php) to be accepted.

[David]

Only the search.php script can write to the log file (external web sites can't write files to your server's hard drive).

Maybe you are getting bots (and humans) entering in search terms on your site that look like rubbish? Automated penetration testing is one common source of this.
You can filter these people out from your site, but our experience is that it is a fruitless task. You block a bunch and the next day there are a bunch more.

[Tapio]

Just FYI.
I successfully blocked a bot that was hammering a prepared searchlink somewhere in the internet and thus filling my searchlog with rubbish.

The trick was to include a (compressed) timestamp in search string, and if the timestamp is either missing or is older than few seconds, the query will be ignored. This forces the user to work on the search template provided. From possible manual misuse I'm not so worried about.

For this, I added some JS and PHP in 'search.php'. The solution and realization may not be optimal, but it worked for me. If anyone is interested of details, the solution is attached (with highlighted extracts).

Attachment (pdf): search.php with timestamp.pdf

[David]

Nice. Might still filter out a few real humans however. e.g someone making a bookmark and coming back to it the next day.

[Tapio]

The deficiency where someone is using an expired bookmark can be mostly overcome by prefilling the search form with query terms from the url - if this is expected to be an issue. Then only, what the user has to do, is to press 'Enter' (submit).

Modified example in 'search.php' (pdf): search-php w timestamp and q-prefill.pdf