Skip to content

dev01 computer setup

Tom Purucker edited this page Mar 6, 2018 · 27 revisions

Creating a qed local development environment

This page is intended to describe the basic setup process for a development machine, either an EPA Windows or Mac.

1) github

get a github account if you do not have one

  • create a github account if you do not have one
  • get yourself added to the quanted group

Install github for desktop

Install git for command line

  • windows command line git commands install git-scm
  • git-scm
  • git comes with xcode command line tools additional install

Caching git credentials on windows

git config --global credential.helper wincred

Caching git credentials on linux (9000 sec = 2.5 hours)

git config credential.helper 'cache --timeout=9000'

2) Install python via anaconda

  • get os 64-bit dependent installation files
  • install python 3.X via anaconda at anaconda
  • default install location on windows @ C:\Users\lan___id\AppData\Local\Continuum\Anaconda3 (or2) (hidden on epa machines)
  • default install location on mac @ ~/anaconda/bin/python

Set windows path (for Windows)

Select Start, select Control Panel. double click System, and select the Advanced tab. Click Environment Variables. ... In the Edit System Variable (or New System Variable) window, specify the value of the PATH environment variable for pythonpath to C:\Users\lan___id\AppData\Local\Continuum\Anaconda3

3) Install pycharm

4) Clone qed project (aor some qed_* subproject) from git

clone from git from parent of where you want to have source
you may want to use your elevated privileges to create a git directory on
the root drive of your machine, for example c:\git

git clone https://github.com/quanted/qed.git
git checkout current_dev_branch # (usually dev)

5) Sideload supplemental data

SAM data available from dropbox https://www.dropbox.com/home/Preprocessed

also from aws console: https://s3.console.aws.amazon.com/s3/buckets/sampreprocessed/?region=us-east-1&tab=overview view data: http://sampreprocessed.s3-website-us-east-1.amazonaws.com/

pip install awscli
aws configure
# download sam inputs from aws s3    
aws s3 sync s3://preprocessed .

6) Database support

install mongodb

install redis

  • prerequisites: download and compile redis for mac/linux
  • windows: download redis
  • celery conda/pip install described in pycharm integration

7) install docker (community edition)

8) bash configuration

modify .bashrc in home directory

modify prompt

export PS1="\A \w \\$ \[$(tput sgr0)\]"

PyCharm Setup:

  • Create new 'Run/Debug Configuration' using 'Python'
  • Set script to 'C:\PATHTO celery.exe in env'
    • example: 'C:\Program Files\Anaconda3\env\py36_qed\Scripts\Celery.exe'
  • Set Script parameters to: '-A tasks worker -Q sam -loglevel=debug -concurrency=1 -n sam_worker'
  • Set Working Directory to 'C:\PATHTO celery.exe in env'
    • example: 'C:\Program Files\Anaconda3\env\py36_qed\Scripts\Celery
  • Save/Apply and Run celery (trying to run in debug mode fails)

Additional Info:

  • sample code can be found in quanted\qed_ubertool sam branch
  • In pycharm with run flask_cgi.py and django in debug mode, celery can be run and watched
  • QED celery process diagram

Other useful installs

platform-independent

  • gdal
  • google earth engine
  • install r
  • install rstudio
  • jetbrains webstorm
  • sublime text
  • db browser for sqlite
  • postman
  • google app engine
  • dropbox
  • kindle
  • microsoft office
  • docker
  • google drive
  • gitter
  • slack
  • chrome
  • skype
  • spotify
  • anaconda navigator

mac extras

mac finder setup

  • finder -> preferences -> show all filename extensions
  • finder -> configure favorites (macintosh hd, desktop, applications, downloads, user, dropbox, git)
  • finder -> view -> show path bar
  • show hidden files

iterm2 setup

  • iterm2 for mac
  • git clone https://github.com/mbadolato/iTerm2-Color-Schemes.git
  • create new: preferences -> profile -> ubuntu
  • add github
  • linked to iterm2 color scheme
  • link colors from github pull
  • preferences -> general -> native full screen
  • preferences -> appearance -> show tab even when there is only one
  • preferences -> profiles -> general -> working directory and send text at start ls -al

cygwin configuration (probably will be killed by windows 10)

Clone this wiki locally