Internal server error on URLs containing accents

[ian_hopkinson]

This generates an internal server error: https://newsreader.scraperwiki.com/actors_of_a_type?uris.0=dbo:Person&filter=J%C3%BCrgen

[Peter Waller]

ERROR:app:Exception on /actors_of_a_type [GET]
Traceback (most recent call last):
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/newsreader/app/views.py", line 70, in run_query
    current_query = assemble_query(query_to_use, query_args, page)
  File "/opt/newsreader/app/views.py", line 86, in assemble_query
    current_query = query_name(**query_args)
  File "/opt/newsreader/app/queries/actors_of_a_type.py", line 15, in __init__
    super(actors_of_a_type, self).__init__(*args, **kwargs)
  File "/opt/newsreader/app/queries/queries.py", line 81, in __init__
    self._make_filter_block()
  File "/opt/newsreader/app/queries/queries.py", line 154, in _make_filter_block
    self.filter_block = '?filterfield bif:contains "{filter}" .'.format(filter=self.filter)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

[ian_hopkinson]

For local testing:

http://127.0.0.1:5000/actors_of_a_type?uris.0=dbo:Person&filter=J%C3%BCrgen

[Steven Maude]

The Python errors are resolved by 5b629dc (in a branch for now).

Now the problem is that for a query like:

SELECT (?filterfield AS ?actor) (COUNT(DISTINCT ?event) AS ?count) ?comment
WHERE {
  ?event sem:hasActor ?filterfield  .
  ?g1 dct:source <http://dbpedia.org/> .
  ?g2 dct:source <http://dbpedia.org/> .
  GRAPH ?g1 {
    ?filterfield a <http://dbpedia.org/ontology/Person> .
  }
  GRAPH ?g2 {
     ?filterfield rdfs:label ?_label . ?_label bif:contains "Jürgen" .
  }
  OPTIONAL { ?filterfield rdfs:comment ?comment }
}
GROUP BY ?filterfield ?comment ?g2
ORDER BY desc(?count)
OFFSET 0
LIMIT 20

the error is

java.io.IOException: Query evaluation failed after 7 ms Virtuoso 37000 Error XM029: Free-text expression, line 0: Invalid character in free-text search expression, it may not appear outside quoted string at à SPARQL query: SELECT (?filterfield AS ?actor) ... GRAPH ?g2 { ?filterfield ?_label . ?_label bif:contains "Jürgen" . } ... Failed query: SELECT (?filterfield AS ?actor) ... GRAPH ?g2 { ?filterfield ?_label . ?_label bif:contains "Jürgen" . } ... 

Doesn't matter whether you URL escape or not, the result is the same. I can't even get this to work in browser by going directly to a query URL. Either we need to do something magic to make the query work (that I can't figure out) or maybe they don't support Unicode characters in Virtuoso? (There are some specific UTF-8 settings, I think and I would have guessed with it being an international project that they'd be aware of those...) Have emailed Marco on the mailing list about this.

[Steven Maude]

This is now a separate problem (how we correctly quote terms), so marking as resolved.

[Steven Maude]

Reopening because it still happens on newsreader.scraperwiki.com, though it doesn't happen locally :(

[Steven Maude]

/var/log/upstart/newsreader.log

ERROR:app:Exception on /actors_of_a_type [GET]
Traceback (most recent call last):
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/newsreader/.local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/newsreader/app/views.py", line 124, in run_query
    query_args = parse_query_string(request.query_string)
  File "/opt/newsreader/app/views.py", line 74, in parse_query_string
    print query_string
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 26: ordinal not in range(128)

This is because it's trying to print the query string... and not being able to handle the character.

[Steven Maude]

Fixed by 9d553e3; phew!

[ian_hopkinson]

We should add a passing test for this since we managed to accidentally revert the fix

[ian_hopkinson]

Need to add a test for this since we accidentally reverted.

[ian_hopkinson]

Fixed by 7c8ebc8 (although I did not force a failing test)