somlor
11-14-2009, 02:26 PM
I have a kickass plan for a site with hundreds of thousands of posts, scraped from several other sites each that update regularly.
I plan to run the scraper to update my site about once a month, but with this much data I really don't want duplicate posts.
Nick, I was just on the webinar (thanks for that) and you mentioned I can have my CSV data replace/update existing posts by using a unique field ID. That's really cool. I'm guessing what I'll do is have my scraper connect to a MySQL database, check for a matching record, if it finds one update it there, otherwise add a new a record. Then once a month I'll export all new and/or updated records out to a CSV with the ID column to prevent dups.
I'm sure you've got a pile of ideas and feature requests but just tossing this out there. It'd be pretty awesome if I could completely avoid the whole export to CSV process (which can get messy) and just hook WFR directly to my database, have it use the MySQL columns just like it parses CSV files, and then I can say hey this is my ID column, update my site once a month.
Sean
I plan to run the scraper to update my site about once a month, but with this much data I really don't want duplicate posts.
Nick, I was just on the webinar (thanks for that) and you mentioned I can have my CSV data replace/update existing posts by using a unique field ID. That's really cool. I'm guessing what I'll do is have my scraper connect to a MySQL database, check for a matching record, if it finds one update it there, otherwise add a new a record. Then once a month I'll export all new and/or updated records out to a CSV with the ID column to prevent dups.
I'm sure you've got a pile of ideas and feature requests but just tossing this out there. It'd be pretty awesome if I could completely avoid the whole export to CSV process (which can get messy) and just hook WFR directly to my database, have it use the MySQL columns just like it parses CSV files, and then I can say hey this is my ID column, update my site once a month.
Sean