I did that during the height of COVID, when my household was only going to the store once a month. Imperfect Foods was how I got fresh produce in between those trips.

It’s a public servants thing–the public wants to know what they’re paying for, so public servant salary records are public.

Various websites compile this information from the various state and federal sources. It’s wicked easy to find information on, say, every public servant with the title “librarian” in Fake County, Kentucky.

Knowing their full name, you can look up their home ownership records in the county real estate or tax databases and ta-da, you know where they live. You also know if they work part-time at a different public library, so that’s convenient for stalking purposes.

Edit: not that I think it’s a good thing. It’s creepy as all get out. If we have to post salaries, I’d much rather they be anonymized like on Glassdoor.

Edit2: and these lists do get used for political ickiness. There’s an anti-union group that mails out helpful tips on how to save money–leave your union. They even provide a “I want to leave” postcard addressed to your union leadership for you to sign, pre-filled-in with your info.

I’m a librarian. I also work with members of the public, some of whom do not share my understanding of reality. My information is still public because I’m a government employee.

Same. Feeling pretty good about using Anubis instead of Cloudflare for our dinky systems.

I wonder if I can get Facebook to give me some of that sweet, sweet cash for the inconvenience of telling them to bugger off…

That’s hardly anything. Facebook has a bot accessing my server’s robots.txt multiple times a second. (My robots.txt used to say “Facebook bot go away” but now I just respond 404 to any requests from the Facebook bot. Pretend I said that all technical and stuff, it’s 2 am and I ought to go to sleep.)

Give Hey Japan a look. I’m a noob, but it seems friendly.

I’d say it depends on how much the license costs vs how the service costs.

The analogy that comes to mind is old cemeteries (YMMV, this is from a New England perspective). People buy a grave and expect to occupy it forever. This is a problem for cemeteries because a cemetery will eventually run out of graves to sell. The sales of graves goes towards the upkeep of the cemetery. Once there’s no more space, there’s no more sales, and there’s no more income for upkeep.

Some cemeteries get around this by reusing graves. You rent a grave for, say, 20 years and after 20 years of occupancy your next of kin is asked if they’d like to renew your subscription.

Other places charge a much higher upfront fee and invest it, using the interest to pay for ongoing maintenance.

Other places just abandon the cemetery and let it grow over with weeds.

And the Wii can be Linuxed, if you get creative enough. Source: former roommate did it

Bots lie about who they are, ignore robots.txt, and come from a gazillion different IPs.

Odd. I just tried

curl https://www.scrapingcourse.com/cloudflare-challenge

and got

Enable JavaScript and cookies to continue

I’m clearly not on the same setup as you are, but my off-the-cuff guess is that your curl command was issued from a system that cloudflare already recognized (IP whitelist, cookies, I dunno).

Anyways, I’m reading through this blog post on using cURL with cloudflare-protected sites and I’m finding it interesting.

Last I checked, cloudflare requires the user to have JavaScript and cookies enabled. My institution doesn’t want to require those because it would likely impact legitimate users as well as bots.

It’s also a huge problem for library/archive/museum websites. We try so hard to make data available to everyone, then some rude bots come along and bring the site down. Adding more resources just uses more resources–the bots expand to fill the container.

Last time I went to a new-to-me grocery store that I didn’t have a loyalty card for, I just used [local area code]-867-5309. Jenny hasn’t failed me yet.