If you have installed XQuartz, you must first tell Mac OSX where X11 is – it’s under Utilities. Then you can navigate to the Coot package (I found it in /applications). Show Package Contents > Contents > Mac OS and double-click Coot. A Coot logo will appear in the dock. You can right-click this logo and use the option Keep In Dock. Voila!