query with / sign

[Leszek Piatek]

Trying to search: “https://facebook.com“

piler-searchd-1  | /* Wed Jan 18 16:21:08.676 2023 conn 831437 real 0.029 wall 0.029 found 0 */ SELECT id FROM piler1 WHERE MATCH(' https://facebook.com') ORDER BY sent DESC; /*error=index piler1: syntax error, unexpected '/' near '//' */

Searching escaped string works: “https:\\/\\/facebook.com“

piler-searchd-1  | /* Wed Jan 18 16:39:14.103 2023 conn 833620 real 0.001 wall 0.002 found 646 */ SELECT id FROM piler1 WHERE MATCH(' https:\\/\\/facebook.com') ORDER BY sent DESC;

Do we have more signs to escape?

Do you have idea how to fix it or it’s a feature? :)

[Janos SUTO]

The parser splits the URLs, see it with pilertest. How about simply

 body: facebook.com

or more precisely:

body: __URL__facebookXcom

‌

[Leszek Piatek]

But You can type / into search box and there is no error displayed or any escaping done before sending query to sphinx/manticore . Isn’t this a issue? Try to search any url with path or just any string with / in it. I think we should escape query before sending it to searchd…

On UI side there is error 500 in console log and no results are returned.

‌

[Janos SUTO]

The slash (/) character is a separator. If you carefully check the pilertest output I don’t think you can find any / characters in it. Thus manticore doesn’t even see anything having (/). Therefore you won’t find anything if searching for (/). Btw. as you can see, manticore doesn’t even like the slash character in the query.

[Leszek Piatek]

Do we have any other special characters that should be ommited in queries?

[Janos SUTO]

You shouldn’t use any special characters that are not meant to instruct manticore like somewo* or “some phrase” or any operators. The tokens that are indexed are sanitized and special characters are removed.

[Leszek Piatek]

What do You think about escaping / in query before sending it to manticore like `$query = str_replace('/', '\\/', $query);` ?

‌

[Janos SUTO]

I still don’t understand your obsession about the slash (/). I told you before: the slash character is not in the text that is sent to be indexed by manticore. Thus no matter if you escape it or not all queries containing the slash will fail.

[Leszek Piatek]

Janos excactly, all queries will fail and me as an end user do not know why? Do You think error 500 is ok?

‌

‌

slash is used widely in URL’s so its quite common input as a query some, path etc?

PS: I’m searching HTTPS not HTTP maybe this is the case that You are not understand because it’s not translated to __URL__facebookXcom ?

piler-searchd-1 | / Tue Jan 24 20:47:55.108 2023 conn 1898744 real 0.000 wall 0.000 found 0 */ SELECT id FROM piler1 WHERE MATCH(' https://facebook.com')) ORDER BY sent DESC; /*error=index piler1: syntax error, unexpected '/' near '//facebook.com' */

‌

[Leszek Piatek]

Ouh and query with escaping is not failing, try to search facebook.com\\/ as below:

‌

[Janos SUTO]

Great, if it works for you, I’m happy with that.

[Leszek Piatek]

Janos but this is a bug, piler is not informing end user about error in his input query, there should be information that / is forbidden or more general error if there is 500 returned from search-helper.php - something is wrong with your query, please make sure your query do not contain / or eny other forbidden character?

Why excactly You don’t want to escape query before parsing and sending it to manticore?

[Janos SUTO]

Right, an 500 status code is definitely not acceptable. I’ve just added a few commits to the master branch, please apply them, and it should be fine by now.

https://bitbucket.org/jsuto/piler/commits/7519da67438c9a62f691b9fb1a96248fc3e6cde3

https://bitbucket.org/jsuto/piler/commits/b463da69fa17bace7f8a7a080948bf1dc507ef25