fix: Custom query validation throwing error with sql files ending with semicolon(;)

[celiajmd]

Closes issue #565

• Remove the semicolon if this semicolon is at the end of the string, or only '\n' and/or spaces is after it.

[celiajmd]

/gcbrun

[nehanene15]

Couldn't we just do an query = query.rstrip(';')? Or would there be some functionality missing if we did that

[Raniksingh]

Both rfind and rstrip does the job. If we need to use it for including complex scenarios then rfind has that flexibility, as it gets the index values which can be used later. I don't foresee any such use case though.
Happy to have more thoughts on this and agree to Neha's point to make it simpler and flexible.

For E2E : We have used 'split' function which basically splits the input query and removes semicolon.

[celiajmd]

Thanks for the second thought!

I tired query = query.rstrip(';') in the first place, but it still gives the error: google.api_core.exceptions.BadRequest: 400 Syntax error: Expected ")" but got ";" at [2:11]

The test query is:

SELECT * FROM `celiaji-dvt-dev.source.trips1` 
LIMIT 1000;

When using a ValueError to print out the query after the

            raise ValueError(
                "query is. "
                f"query is: {query}"
            )

We can see in the error message, the semicolon is still there:

raise ValueError(
ValueError: query is SELECT * FROM `celiaji-dvt-dev.source.trips1` 
LIMIT 1000;

One assumption is that the query string has multiple lines, and the semicolon only appears at the last line, so that the rstrip(';') hasn't been able to detect or delete it in this case. But, I did test rstrip(';') on the query string alone, and it works. I am not sure if it is because there are other functions in DVT which may interfere the result.

So I switched to rfind(). rfind() can make sure to find the last meaningless semicolon, by checking if the right side of this semicolon are all just spaces and enters.

I didn't choose split() is because I wanted to save the semicolon which may be meaningful in a query string, such as select * from table a where text = "abc;"

[nehanene15]

The python file.read() method automatically appends a newline at the end of the string returned. Therefore, we should expect a newline and trim the newline/whitespace and semicolon accordingly:
query = query.rstrip(';\n'))

Using rstrip will ensure we are only removing trailing semicolons and not those in the query

[nehanene15]

Wondering if we could make this even simpler.

[celiajmd]

/gcbrun