The link syntax supported by Robot is [url|title]. This problem occurs because the link formatter matches the opening [ with the first closing ] and thus in [url|my [other] awesome title] the URL is url and the title is my [other.

The problem could be fixed by making the formatter smarter. It could handle opening [ in URL and title parts so that the next closing ] would be considered a literal value and not closing the whole link. It would require changing the current regexp based approach to a smarter parser. It wouldn't be that complicated, but the change isn't trivial either. There's also a problem with handling unbalanced bracket usages like [url|tit[le].

I personally don't consider the problem so bad that fixing it would be worth the effort. That said, if someone else is interested to fix it, I'm happy to review and merge PRs. One reason I don't consider this too high priority is that I think we should move from our custom documentation format to Markdown in the future. We are planning to take the first steps to that direction in RF 7.3 (#5304) and hopefully can extend to support to suite documentation and metadata in the future. After that enhancing our own syntax makes pretty much no sense.