Programming Spiders, Bots, and Aggregators in Java [With CDROM]

by Jeff Heaton

The content and services available on the web continue to be accessed mostly through direct human control. But this is changing. Increasingly, users rely on automated agents that save them time and effort by programmatically retrieving content, performing complex interactions, and aggregating data from diverse sources. Programming Spiders, Bots, and Aggregators in Java teaches you how to build and deploy a wide variety of these agents-from single-purpose...

• Format: Paperback
• Language: English
• ISBN: 0782140408
• ISBN13: 9780782140408
• Release Date: February 2002
• Publisher: Sybex
• Length: 516 Pages
• Weight: 2.25 lbs.
• Dimensions: 1.2" x 7.5" x 9.0"

Customer Reviews

Create a Object Oriented Bot Package Step by Step

I use this book as a supplement to a class that I teach, as it gives the students the necessary stills to programmatically spider, and generally access, information on the Net.As some of the other reviewers point out, this book does center around the creation of a "bot package". However, I see this as one of the book's greatest strengths. The author explains step by step how to take basic concepts, continually build upon them, progressing onward to more complex spiders and bots. Specifically:1. Create an advanced HTTP object that overcomes many of the shortcomings of the one which is built into Java. (namely cookie support, referrer support, HTTP authentication, and more)2. Add forms/page processing on top of the HTTP object. You are shown step by step how to process the data you collect from step 1.3. Create a bot that wields the page/form processing created in step 2.4. Create a spider, that, using steps 1-3, can access pages across an entire site.5. Expand the spider to support thread pooling and a JDBC database.Rather than providing a bunch of disjoint code samples, like many books do. The author guides you step by step through the above path, revealing the techniques at every step. For the reader who does not care about the intricate nature of bot programming, sadly, some of my students. You can skip to the API documentation and get right onto creating your own bots. You can also download updated versions of the "bot package" from the author's site. I actually did this before buying the book. The downsides to the book are the example programs use of GUI's. I would rather every example had been straight console, the GUI only gets in the way, for a book targeting bot programming. Also the author very annoyingly putting an underscore in front of every class-instance variable, which gives some of the code something of a C++ look I suppose.If you are already programming bots and spiders of your own, I don't think you will get much more from this book than you are likely already doing. But for someone who wants to get started in this exciting area, there is nothing else like it, and I highly recommend it.

happy

Visual Cafe produces the Swing so one can view the examples from the book. So what? When beginning to program with HTTP protocols, it's easy to enter incorrect methods and parameters that lead to dead-ends and frustration. As I learn about and use the Heaton API, I am pleasantly surprised with the methods available and how easily they're implemented and that they lead to success.The source code is included on the CD with updated versions at the Heaton Website.

A great example of how to present highly technical material

I've read MANY technical books over my 23 year career in IT and would say this this book is one of my top 3 favorites. Although I'm a novice Java programmer, I never felt lost within this material. The author (who teaches at a community college) is obviously skilled in the proper presentation of difficult material, a talent which is lacking amongst most technical authors. I suspect his experience teaching Java and other classes at the college goes a long way in increasing his ability to communicate well.Each chapter initially exposes you to the goals within, then covers underlying technologies, when necessary. After showing the Java code which represents the current project, he drills into the meat of the code with the "under the hood" sections. At first, some of the code was confusing to me, but by following the detailed explanation, I was able to clear everything up and feel confident that I knew what was going on. At the end of each chapter, the summary gives a nice review which helped me to "lock in" what I had learned and to put the chapter material into perspective.The code samples were well written and they all compiled and ran fine (compiled code is included). The spider libraries on the CD are designed intelligently and easy to build on.I'm very happy that I chose to read this book and found it to be very enjoyable, considering it's technical nature. I definately feel that I'm equipped to do creative bot coding in Java. Also, my knowlege of Java was generally deepened.

Incredible Book!

This book is simply one of the best computer books I have ever read! The author doesn?t just cover programming like a lot of programming books do, he also explains how technology related to the subject matter works. Some of the related subjects he covers are socket programming, the HTTP protocol, form submission, cookies, multi-threading, JDBC, and XML. The coverage of these extra topics is not very in-depth, but for the average Java programmer, he provides enough information to fill in some gaps and tie it all together, and that?s where the strength of this book lies. Another thing the author does well is build on each chapter, pacing it quickly enough to not bore but not so fast as to overwhelm the reader. And finally, the author provides sections in most chapters that explain in detail how the classes used in that chapter function.The book is very well organized and laid out in a logical manner. The book?s topics start at simple Java socket programming, progress through HTTP and HTTPS socket programming, then into HTML parsing and form posting, to dealing with other file/data formats, using cookies, then into spiders and multi-threaded spiders, then bots, then aggregators. He finishes up by discussing the ethics of using these constructs to access sites and the possible unintended consequences of doing so, and a brief discussion of how these constructs may fit into the web services arena.A major plus in this book is the companion cd, which in addition to containing the source code for all the examples in the book has the classes already compiled, as well as a copy of the Jakarta/Tomcat server, since some of the examples use JSPs. The companion cd was supposed to contain a JDK as well, but it was left off in error. The software provided on the cd is reusable, and I will be integrating it into a project in the near future.All in all, this is an outstanding book, and worth more than the money paid for it. The book does assume a few things: you have to know at least the basics of Java, some familiarity with the Internet, and some familiarity with JSPs would be a nice to have. This book is a must-read for advanced beginner and intermediate developers, but expert Java developers should think twice as they are likely already familiar with most of the material.

Great Book!

This just came out March 2002. I'm very impressed. It's a complete intro to Bots in Java. The CD has a bot.jar package that will really come in handy. It solves the problem of cookies, user authentication, and the referrer variable missing in javax.net.* Maybe someday Sun will fix those problems. The book covers useful stuff like posting forms, frames problems, cookies, SSL, authentication, parsing XML, QIF, and CSV formats etc. It also has several complete expamples. This book uses J2SE 1.3. My only complaint is it does not have examples or talk much about artificial intelligence or handling JavaScript although it does mention Rhino the JavaScript engine written in Java.Neil