Add fixed delay in between DDL changes to accommodate node time drift

[sebbonnet]

Each DDL change should be separated by a fixed delay to ensure we cater for time drift between the nodes: delay >= max node time drift

When performing DDL changes, cassandra is always using the node timestamp when announcing alter table statements or other DDL changes regardless of whether a custom timestamp generator is provided when creating the session. As each DDL change can be executed on a different node, we need to wait long enough to ensure upgrades are timestamp ordered across all nodes.

[oliverlockwood]

@sebbonnet It would be good to fix this issue, thanks for raising. Alongside adding this workaround in cqlmigrate, is there anything we can feed back into open-source Cassandra to suggest a fix on their side?

[sebbonnet]

@oliverlockwood I've fed back the issue here https://issues.apache.org/jira/browse/CASSANDRA-11874, but I doubt it will get fixed given that

• client timestamp are also ignored in Lightweight transactions (LWT)
• there is a strong resistence to remove server-side generated timestamp - see https://issues.apache.org/jira/browse/CASSANDRA-6178

[oliverlockwood]

@sebbonnet 👍

[emperorz]

Just a thought: you could choose a node (IP address) and whitelist it using the whitelist round-robin policy.

Should allow you to guarantee single node DDL application. See: https://datastax.github.io/python-driver/api/cassandra/policies.html#cassandra.policies.WhiteListRoundRobinPolicy