AUGS Doctor Directory Web Scrape

Starting URL: http://www.voicesforpfd.org/p/cm/ld/fid=81 We’d like to collect some information on doctors listed on the website above in two CSV files. Visit the URL above. For “State,” select “Alabama” and click “Search.” You will receive a page of results. For this project, we need to get all results for all U.S. States. We do not want results for Canadian provinces or any other country. For all states except for New York and California, results are available on one search page. For New York and California, the first 100 records can be accessed with a simple search. Results are ordered by last name A-Z, so the remaining records can be accessed by searching on last name “m”, “n”, etc. through the alphabet until all records are identified. There should be approximately 900-1,800 records in total. We want to collect the following information, when available, in a CSV file with one row for each doctor or healthcare professional: id (incrementing numeric id assigned by your script) email (if available) name suffix (Jr., etc.) title (MD, DO, etc.) phone fax (if present) addr1 addr2 city state zip country affiliation (i.e. current employer or other professional affiliation) affiliation_url (url for employer) description (job title- for example, “Physician”, located below name) board_certified_fpmrs ( "1" if result has the “Board Certified FPMRS Subspecialist” identifier, "0" otherwise ) This file should be named scrape_AUGS_{date} The first row should contain column names exactly as listed above. See example file [login to view URL] ====================== Each result also has a small icon with a blue “i”. Hovering on this icon reveals several data points including “Areas of Focus/Specialties.” We want to collect a CSV file with a row for each Area of Focus/Specialty for each doctor listed. The file should have the following format: id (incrementing numeric id assigned by your script) scrape_AUGS_{date}_id (the id for this doctor corresponding to the id in your script ) specialty (name of specialty) This file should be named scrape_AUGS_{date}_specialty The first row should contain column names exactly as listed above. See example file [login to view URL] (We have only shown two specialties per user in this example. Your final file should include all specialties.)