Should your site offer job spidering - or not?

Getting jobs from an employer and onto a job board has been a challenge from the earliest days of the industry. Initially, the only way to do so was manually – the employer filled out a web form and submitted the job ad to the job board. Needless to say, this method (which, of course, is still widely used) was time-consuming and provided many opportunities for typos, dropped information, and other inaccuracies.

Needless to say, job boards didn’t like this arrangement – the harder it was to post jobs, the less likely their customers were to buy more job ads than absolutely needed. Employers didn’t like it, for it was tedious, slow, and yet another task for their (under-staffed) HR departments.

Keep reading....

Tags: job, posting, scraping, site, spidering

Comment by Todd Hayton on May 15, 2012 at 9:39am: Interesting article - one viewpoint not mentioned is whether job seekers would prefer one over the other. Personally, as a jobseeker I like the idea of jobs spidered directly from company websites over job postings because it means you don't have to deal with all of the fake job ads that I see pop up on job boards where they won't list the company offering the job. I don't want to have to wade through "Company Confidential" type job ads and frankly as job seeker it always felt a bit like spam. Of course, a job board could filter out such ads, but in my experience many of them don't.

Comment by Shaun Pilfold on May 16, 2012 at 12:36am: It can take a lifetime to use "then along came a spider" in a conversation, well done Jeff ;)

Comment by Shankar Srinivasan on May 16, 2012 at 7:21am: Like it or not, Job wrapping (spidering) will be a service we Job boarders will have to offer in the future if we want more Jobs coming our way. Be it manually entering Jobs from a company website or parsing it automatically, it is a necessity.

Comment by Todd Hayton on May 16, 2012 at 8:00am: Makes me wonder if there would be a b2b market for job-scraping as a service...

Comment by Jeff Dickey-Chasins on May 16, 2012 at 8:30am: Todd, I know there are several companies that do scraping, including several of the job board software companies.

Comment by Alexander Gutin on May 18, 2012 at 2:12pm: Jeff, I agree 100%, your reasons listed in the article are exactly why we started offering this service. Employers are busy and the ones that have a bunch of jobs to post certainly don't want to have to do this over and over.

Unfortunately, there is no standardized format for these jobs that all job boards have agreed on, so until that happens, job spidering is absolutely a necessity for job board owners. Although these spiders obviously add cost to the operation fo the job board, they should be offset by the increased number of postings they'll be making from the employer whose jobs are being spidered. In other words, job spiders should easily pay for themselves.

Good post Todd!

Comment by Jeremy Haskell on May 21, 2012 at 11:42am: Has there been much research done regarding the time frame of scraping/spidering? We were offering a file transfer protocol, that was deeply flawed due to the lack of support on the client end (sending incomplete files, etc) and are considering moving to a scraping solution.

One problem some clients have brought up has been the delay between when the site is scraped, and when the jobs will show up on our site. Anyone else face this problem? I haven't been able to locate a "real time" scraping protocol, and would be very much interested if anyone has information about this.

Comment by Jeff Dickey-Chasins on May 21, 2012 at 11:54am: In my experience, companies expect to see changes within 24 hours or less. So if you're spidering a site, you need to achieve at least that level of turnaround. Most spidering services I've seen are much faster.

Comment by Jeff Dickey-Chasins on May 21, 2012 at 1:48pm: I know that JobMount offers it - they call it SpiderMount: http://www.jobboardmount.com/cm/job_spider

Comment by Todd Hayton on May 21, 2012 at 1:54pm: "One problem some clients have brought up has been the delay between when the site is scraped, and when the jobs will show up on our site. Anyone else face this problem?"

Seems odd - I could understand a delay between scraping sessions (to reduce load on the servers of the companies being scraped), but not between scraping & upload. The only thing I would think that would occur between scraping the jobs and having them show up is 1) indexing the jobs so that you can search for keywords against the job descriptions and 2) geocoding the locations in the jobs so you can query for jobs by location/distance. Neither of those should take very long.

Should your site offer job spidering - or not?

You need to be a member of Job Boarders to add comments!