The setgid bit works in two ways, one for files and one for directories.
The setgid bit for files will force when a file is executed to set its group ID to the GID that of the binary instead of the user that is running it. This mode has no effect for files that are not executable. To a lesser extent than SetUID this can lead to security issues when the group in question has some files that would be allowed to be read or written when otherwise not permitted. There are very limited uses for this feature these days and its use is discouraged.
SetGID directories have a much more benign behavior. When this bit is set on a directory all filesystem creations underneath that directory will inherit the group from the directory. UMIACS policy is to keep users' default UID and GID. So for users who wish to have certain directory trees have consistent groups the use of SetGID directories is useful. The SetGID directory bit will force all files created under that directory to have the same GID. Any directories created will also be given the same GID and the SetGID bit so that the same policy is applied at the next level of directory down.
Before you we give an example of how to use the sticky bit please see the documentation about umask. SetGID will only work on the GID and the setting a sticky bit and will not change any other permissions. By default the umask is 022 and is a bit too restrictive for the use of SetGID effectively. (002 or 007 will be the appropriate setting depending if you want Others to be able to read/execute or not)
First an example of what my default group is and what other groups i belong to:
[derek@novelty ~/staff]$ id uid=2174(derek) gid=22174(derek) groups=15114(umadmin),15116(vnodedisplay),22174(derek)
Now i have created a directory called staff that I will want to share with the umadmin group (which is not my default group)
[derek@novelty ~/staff]$ ls -la . drwxr-xr-x 2 derek derek 96 Jun 16 11:34 .
I am going to set the new group to umadmin and set rwxrwxr-x permissions. Then finally i will add the SetGID bit.
[derek@novelty ~/staff]$ chgrp umadmin . [derek@novelty ~/staff]$ chmod 775 . [derek@novelty ~/staff]$ chmod g+s .
Now you can see we have a directory that is correctly set up,
[derek@novelty ~/staff]$ ls -la . drwxrwsr-x 2 derek umadmin 96 Jun 16 11:34 .
I am going to use umask of 002 since I want to allow others to read the files.
We are going to now create a test file and test directory,
[derek@novelty ~/staff]$ touch test [derek@novelty ~/staff]$ mkdir testdir
Now as you can see we have the correct groups and permissions being used to create both files directories,
[derek@novelty ~/staff]$ ls -la drwxrwsr-x 3 derek umadmin 96 Jun 16 12:58 . -rw-rw-r-- 1 derek umadmin 0 Jun 16 12:58 test drwxrwsr-x 2 derek umadmin 96 Jun 16 12:58 testdir