We didn't go [because you were] late.

Correct!

The grouping of the words "because you were" is not a complete thought.