Colleagues,

 

Might one, two, or more of you have advice on software for web scraping?  My
particular and immediate interest is in one-time scraping of modest amounts
of text data from government websites in the United States.  Of course, if I
like this hammer..

 

I have found only brief anecdotal comments about how to do it in political
science methods papers, and even in papers that obviously have carried out
such a task.  

 

Much software of this ilk is touted on the web, but rare is the online paper
that reviews more than one piece of software.  And the most useful, if still
limited, papers of the latter sort that I have found are at:

 

http://lethain.com/an-introduction-to-compassionate-screenscraping/

 

and

 

http://www.notprovided.eu/six-tools-web-scraping-use-data-journalism-creatin
g-insightful-content/

 

From having read those sources and the text on a number of different
software options, I am led to believe that such software varies by how "user
friendly" it is, how technically sophisticated it can be, and how much user
specified programming is necessary to employ a given software package.
Thus I, and perhaps other readers of this list might appreciate advice from
those of you who have used such software about several options across the
range of those attributes.

 

Kim

 

Kim Quaile Hill

Cullen-McFadden Professor of Political Science, 

Presidential Professor for Teaching Excellence, and

Eppright Professor in UndergraduateTeaching Excellence

Department of Political Science

4348 TAMU

Texas A&M University

College Station, TX 77843-4348

ph. 979/845-8235

fax 979/847-8924

e-mail:   <mailto:[log in to unmask]> [log in to unmask]

 


**********************************************************
             Political Methodology E-Mail List
   Editors: Ethan Porter        <[log in to unmask]>
            Gregory Whitfield   <[log in to unmask]>
**********************************************************
        Send messages to [log in to unmask]
  To join the list, cancel your subscription, or modify
           your subscription settings visit:

          http://polmeth.wustl.edu/polmeth.php

**********************************************************