upgrade PostgreSQL from version 9.5 to version 9.6 without losing data

Assuming you've used home-brew to install and upgrade Postgres, you can perform the following steps.

1. Stop current Postgres server:
   launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
2. Initialize a new 9.6 database:
   initdb /usr/local/var/postgres9.6 -E utf8
3. run pg_upgrade:

   pg_upgrade -v \
       -d /usr/local/var/postgres \
       -D /usr/local/var/postgres9.6 \
       -b /usr/local/Cellar/postgresql/9.5.5/bin/ \
       -B /usr/local/Cellar/postgresql/9.6.1/bin/

4. Move new data into place:

   cd /usr/local/var
   mv postgres postgres9.5
   mv postgres9.6 postgres

5. Restart Postgres:
   launchctl load ~/Library/LaunchAgents/homebrew.mxcl.postgresql.plist
6. Check /usr/local/var/postgres/server.log for details and to make sure the new server started properly.
7. Finally, re-install the rails pg gem

   gem uninstall pg
   gem install pg

I suggest you take some time to read the PostgreSQL documentation to understand exactly what you're doing in the above steps to minimize frustrations.