Thali uses GitHub’s PRs for code reviews and the UX is not up to the job. There is no good way to track which comments are and are not resolved. There is no good way to see comments when the file they are on has changed. I don’t need much, but I really need that. To solve this problem we want a GitHub based code review tool that is hosted. To this end I reviewed five possibilities, none where great and none really solves the problem. But we’ll at least try out codereviewhub.com.
The tool we want needs to:
- work directly in GitHub,
- be hosted,
- support comments being tracked even when the underlying file is changed, ideally by showing the comment in the context of the change so you can verify if the change is what you wanted,
- support allowing the person who opened the comment to mark when it is satisfied and
- indicated when all comments are satisfied
I’m fairly shocked that GitHub doesn’t natively support this kind of functionality.
The solutions I checked out were:
reviewable.io My first impression was that it has these weird icons and no tool tips to explain what they are or how they work. Also simple things were hard. Such as when I opened up a comment I couldn’t figure out how to close it! I just found the whole UX so off putting I ran away.
gerrithub.io I don’t like their “import” model. I want PRs to just show up, not have to be imported. And the UX really confused me. It seems to want to show each commit in a PR separately rather than display the sum total of changes in a PR in one view. I suspect there is some place where one can find the sum total of changes but I couldn’t figure it out.
codifferous.com It took it more than a full day to import a single repro. That was way too long to wait so I ran away.
www.review.ninja Right off I can’t say I like the interface. Rather than comments being inline to the file they are over to the right which is really annoying when one is viewing the diff on a big screen. In addition if there are multiple commits and a comment is made on an old commit the best it can do is take you to that old commit. It can’t show you the comment in the context of the change. But the real killer for me is that when you click on the file link in a comment in the global comment list it just takes you to the right commit but doesn’t jump the screen to the file and line the comment was made on! That makes the tool useless for big PRs.
codereviewhub.com I want to give cudos to the developer for letting you specify if you want to give his site GitHub permissions for your private repo’s. This is really nice. It lets you test things out without having to expose your (or other people’s) private data. What this site does is add a plugin straight to GitHub’s UX that displays a task list containing all comments made on the PR. If a comment is either check marked or if someone adds a :+1: comment then the comment will be checked off and marked Completed otherwise it will be marked Remaining. That’s it. You can also click on any of the comments in the summary and it will jump to that comment. My only complaint is that to get back to the task list you either have to hit the back key or scroll up. I wish there was some UX chrome that made it easier but that’s a minor quibble. Oh and there is a bug that if you delete a comment that isn’t picked up by codereviewhub.com. Also, this isn’t the ideal solution to our problem since it doesn’t let you see comments on one commit in context when another commit is made to fix it but oh well. I guess we can’t have everything.
For some reason this writeup made me think of Monty Python’s “Run away! Run away!” scene from the Holy Grail. :) For the record, Reviewable now has a full-featured help mode (even better than tooltips), and you can close an empty open comment by clicking anywhere outside it.
Yup, the Monty Python reference pretty much nails it. I’ll try to give reviewable.io another try. I just tried out GitColony and rejected it because of problems with how it tracks comments (it’s easy to create a situation where you can’t find the line that a comment connects to). But I had an awesome conversation about this with Mariano and he said that his folks would have it fixed in the next week or two so I plan on giving them a try again as well. I know lots of folks at [work are using GitHub](https://github.com/Microsoft) for more and more projects and the code review situation is just a disaster. Anything decent would, I suspect, get rapidly adopted.
Yeah, I’ve been seeing a fair amount of interest from Microsoft folks in the last month, and it looks like a few projects are starting to use Reviewable regularly. Let me know how it goes!
Agreed. It is incredibly frustrating to try and figure out what CR comments have been addressed. Wish I could opt out of the new flow.
For the project I work on, http://www.thaliproject.org, we use reviewable.io and have had an awesome experience with it. So we don’t use GitHub’s CR tools at all.