Scraping public data from the web still OK: US court
(2022/04/19)
- Reference: 1650327477
- News link: https://www.theregister.co.uk/2022/04/19/scraping_public_data_linkedin/
- Source link:
Scraping data from a public website doesn't violate America's Computer Fraud and Abuse Act (CFAA), the US Ninth Circuit Court of Appeals ruled on Monday.
The [1]decision [PDF] echoes the appeal's court [2]2019 decision , which upheld a lower court's [3]2017 determination in HiQ v. LinkedIn that web scraping doesn't qualify as accessing a protected computer without authorization.
The case began in California in 2017 when HiQ, an employment analytics firm, filed a lawsuit challenging LinkedIn's legal and technical efforts to block HiQ from copying public profile data from LinkedIn users.
[4]
The district judge hearing the case [5]granted a preliminary injunction to HiQ that barred LinkedIn from interfering with HiQ's data scraping while the case progressed. He decided it didn't make any sense to apply the CFAA – a law that criminalizes accessing a protected computer "without authorization" or in a way that "exceeds authorized access" – to the collection of public data from LinkedIn's website.
[6]
[7]
LinkedIn nonetheless appealed and two years later the Ninth Circuit [8]sided with HiQ and sent the case back to the Northern District of California to be resolved.
Undeterred, LinkedIn appealed to the US Supreme Court. In March 2020, it [9]asked the Supreme Court to review the Ninth Circuit ruling. The company argued that implementing technical barriers to web scraping in conjunction with sending a cease-and-desist letter together should qualify as an authorization mechanism. In effect, the Microsoft-owned social media site wants to have competitive benefits of gated access without the consequences – invisibility to search engine traffic.
[10]
"Under the Ninth Circuit’s rule, every company with a public portion of its website that is integral to the operation of its business – from online retailers like Ticketmaster and Amazon to social networking platforms like Twitter – will be exposed to invasive bots deployed by free-riders unless they place those websites entirely behind password barricades," LinkedIn's attorneys wrote in the company's [11]petition [PDF] to be heard by the Supreme Court.
"But if that happens, those websites will no longer be indexable by search engines, which will make information less available to discovery by the primary means by which people obtain information on the Internet."
[12]US Supreme Court gives LinkedIn another shot at stymieing web scraping
[13]Poetry in lockdown: hiQ to Supremes / Please leave LinkedIn scrape ruling / well enough alone
[14]Welcome to The Reg's poetry corner... hiQ once again / beats LinkedIn on web scrape case / more appeals await
[15]hiQ prevails / LinkedIn must allow scraping / Of your page info
On June 3, 2021, the Supreme Court in a related case, Van Buren v. United States , narrowed the CFAA, which had for years been criticized for failing to define "without authorization" and "exceeds authorized access."
The high court in Van Buren [16]said that breaking terms of service alone does not qualify as "exceeds authorized access" under the CFAA. Yet it left some ambiguity about whether credential-based gating is the only way to determine whether access was "without authorization."
Then two weeks later, the Supreme Court sent HiQ v. LinkedIn back to the Ninth Circuit for reconsideration in light of how Van Buren had reshaped CFAA liability. Now, the appeals court has revisited its earlier decision and come to the same conclusion it did two years ago, albeit bolstered by the Van Buren case.
[17]
"[A] defining feature of public websites is that their publicly available sections lack limitations on access; instead, those sections are open to anyone with a web browser," the Ninth Circuit [18]ruling [PDF] says.
"In other words, applying the 'gates' analogy to a computer hosting publicly available web pages, that computer has erected no gates to lift or lower in the first place. Van Buren therefore reinforces our conclusion that the concept of 'without authorization' does not apply to public websites."
The ruling doesn't resolve the HiQ's dispute with LinkedIn, however. It merely prevents LinkedIn from blocking HiQ's gathering of public data and from making a claim against the analytics biz under the CFAA. Issues related to unfair competition, privacy, and state law have yet to be addressed.
In a statement emailed to The Register , a spokesperson for LinkedIn indicated the company intends to keep fighting in court.
"We’re disappointed, but this was a preliminary ruling and the case is far from over," a company spokesperson said. "We will continue to fight to protect our members' ability to control the information they make available on LinkedIn." ®
Get our [19]Tech Resources
[1] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf
[2] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[3] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[9] https://newmedialaw.proskauer.com/2020/03/11/linkedin-files-petition-to-the-supreme-court-in-hiq-web-scraping-case/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://newmedialaw.proskauer.com/wp-content/uploads/sites/22/2020/03/LinkedInVhiQ-LinkedIn-Petition.pdf
[12] https://www.theregister.com/2021/06/15/linkedin_supreme_court_scraping/
[13] https://www.theregister.com/2020/06/29/linkedin_hiq_supreme_court/
[14] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[15] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[16] https://www.theregister.com/2021/06/03/supreme_court_cfaa/
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf
[19] https://whitepapers.theregister.com/
The [1]decision [PDF] echoes the appeal's court [2]2019 decision , which upheld a lower court's [3]2017 determination in HiQ v. LinkedIn that web scraping doesn't qualify as accessing a protected computer without authorization.
The case began in California in 2017 when HiQ, an employment analytics firm, filed a lawsuit challenging LinkedIn's legal and technical efforts to block HiQ from copying public profile data from LinkedIn users.
[4]
The district judge hearing the case [5]granted a preliminary injunction to HiQ that barred LinkedIn from interfering with HiQ's data scraping while the case progressed. He decided it didn't make any sense to apply the CFAA – a law that criminalizes accessing a protected computer "without authorization" or in a way that "exceeds authorized access" – to the collection of public data from LinkedIn's website.
[6]
[7]
LinkedIn nonetheless appealed and two years later the Ninth Circuit [8]sided with HiQ and sent the case back to the Northern District of California to be resolved.
Undeterred, LinkedIn appealed to the US Supreme Court. In March 2020, it [9]asked the Supreme Court to review the Ninth Circuit ruling. The company argued that implementing technical barriers to web scraping in conjunction with sending a cease-and-desist letter together should qualify as an authorization mechanism. In effect, the Microsoft-owned social media site wants to have competitive benefits of gated access without the consequences – invisibility to search engine traffic.
[10]
"Under the Ninth Circuit’s rule, every company with a public portion of its website that is integral to the operation of its business – from online retailers like Ticketmaster and Amazon to social networking platforms like Twitter – will be exposed to invasive bots deployed by free-riders unless they place those websites entirely behind password barricades," LinkedIn's attorneys wrote in the company's [11]petition [PDF] to be heard by the Supreme Court.
"But if that happens, those websites will no longer be indexable by search engines, which will make information less available to discovery by the primary means by which people obtain information on the Internet."
[12]US Supreme Court gives LinkedIn another shot at stymieing web scraping
[13]Poetry in lockdown: hiQ to Supremes / Please leave LinkedIn scrape ruling / well enough alone
[14]Welcome to The Reg's poetry corner... hiQ once again / beats LinkedIn on web scrape case / more appeals await
[15]hiQ prevails / LinkedIn must allow scraping / Of your page info
On June 3, 2021, the Supreme Court in a related case, Van Buren v. United States , narrowed the CFAA, which had for years been criticized for failing to define "without authorization" and "exceeds authorized access."
The high court in Van Buren [16]said that breaking terms of service alone does not qualify as "exceeds authorized access" under the CFAA. Yet it left some ambiguity about whether credential-based gating is the only way to determine whether access was "without authorization."
Then two weeks later, the Supreme Court sent HiQ v. LinkedIn back to the Ninth Circuit for reconsideration in light of how Van Buren had reshaped CFAA liability. Now, the appeals court has revisited its earlier decision and come to the same conclusion it did two years ago, albeit bolstered by the Van Buren case.
[17]
"[A] defining feature of public websites is that their publicly available sections lack limitations on access; instead, those sections are open to anyone with a web browser," the Ninth Circuit [18]ruling [PDF] says.
"In other words, applying the 'gates' analogy to a computer hosting publicly available web pages, that computer has erected no gates to lift or lower in the first place. Van Buren therefore reinforces our conclusion that the concept of 'without authorization' does not apply to public websites."
The ruling doesn't resolve the HiQ's dispute with LinkedIn, however. It merely prevents LinkedIn from blocking HiQ's gathering of public data and from making a claim against the analytics biz under the CFAA. Issues related to unfair competition, privacy, and state law have yet to be addressed.
In a statement emailed to The Register , a spokesperson for LinkedIn indicated the company intends to keep fighting in court.
"We’re disappointed, but this was a preliminary ruling and the case is far from over," a company spokesperson said. "We will continue to fight to protect our members' ability to control the information they make available on LinkedIn." ®
Get our [19]Tech Resources
[1] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf
[2] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[3] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[8] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[9] https://newmedialaw.proskauer.com/2020/03/11/linkedin-files-petition-to-the-supreme-court-in-hiq-web-scraping-case/
[10] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[11] https://newmedialaw.proskauer.com/wp-content/uploads/sites/22/2020/03/LinkedInVhiQ-LinkedIn-Petition.pdf
[12] https://www.theregister.com/2021/06/15/linkedin_supreme_court_scraping/
[13] https://www.theregister.com/2020/06/29/linkedin_hiq_supreme_court/
[14] https://www.theregister.com/2019/09/09/hiq_wins_linkedin_appeal/
[15] https://www.theregister.com/2017/08/14/hiq_linkedin_bots_scraping/
[16] https://www.theregister.com/2021/06/03/supreme_court_cfaa/
[17] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33Yl4z7skEZ6bE3eRW0VRaLAAAAU4&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[18] https://cdn.ca9.uscourts.gov/datastore/opinions/2022/04/18/17-16783.pdf
[19] https://whitepapers.theregister.com/
Re: site stupidity.
ShadowSystems
I am amused that LinkedIn thinks that their users don't already have the ability to keep their private details private. You're a publicly facing site with public profiles for anyone/everyone to browse to try & locate potential employees or past employees that you want to reconnect with, so how *exactly* is anything they post to the site supposed to be private at all?
Yes scraping is a shitty thing to do, but you don't get to argue for privacy if the whole fekkin' point of your site is to make your members as visible & findable as possible.
Sheesh...
msobkow
"Preliminary ruling"?
From the Appeals Court I believe was mentioned. That is two strikes to permanent rejection; hardly what one would call "preliminary." Instead, they seem to keep adding to their list of reasons you should take a hike!
Where do you intend to try to escalate your case to? The Pope? The Dalai Lama? Directly to the deity of your choice?
Re: Kicking it up the ladder.
ShadowSystems
Exactly. It's already gone to the Supreme Court whom kicked it back down for revisitation, but otherwise refused to hear your plea. To whom are you going to "go over their heads" to? Elvis? Elmo? Foamy the Angry Squirrel?
Granted, they have other bits of the case to argue in court, but the bits about scraping seem to have just been shoved off the table into the dust bin.
Anonymous Coward
Apropos of nothing, Twitter, LinkedIn and Instagram (presumably amongst others) have spent large chunks of the last 12ish months severely limiting and even entirely disabling access for those who are not logged in - aka those who have not ticked the magic compliance tickbox, subject only to the whims of arbitration in some armpit court in some arbitrary state.
The same goes for many significant news organisations who are strengthening their paywalls.
Anonymous/"unauthorized" access is disappearing from much of the web.
Strikes me as similar to the case where browsers were used to look at "hidden" elements on web pages
and use those supposedly hidden elements to gather lots of personal information.
I think this was some stupid US state that thought putting "hidden" would make personally-identifiable information disappear.
I know LinkedIn is part of the Microsoft empire, but even so, I thought they would be a bit smarter than this.
"If you hang your laundry out on the line, everyone will know about your doings."