2

I have built a website which is a directory of venues / places in a country and client has asked me to make URLs more SEO friendly.

There has been some misunderstanding in communication and I have structured URLs of places as following:

www.example.com/place/1234/city-place-name-slug

I thought I have followed best practices but now client is angry with me, claiming that I have reduced "security" of website to zero by using such URLs. I have slug of each entity (place) stored in database, so I verify ID and slug, so increasing id in URL won't result in opening next place in DB, you would have to guess complete slug + id combination.

His argument is, that I should have added one extra random letter to the end of the slug, creating following URL:

www.example.com/place/1234/city-place-name-slug-X

I don't completely understand his security concern, but I think he is afraid bots will crawl website and 'steal' information of places he is manually entering in database.

My question is:

  • does adding one random letter to URL at the end make sense
  • are URLs as I structured too easy for bots to "guess" and steal content of website places
3
  • 1
    If you can get to the page, then it can be crawled/scraped. It sounds like they don't have a clue, have they paid you? Maybe they're punting for a discount. One could readily search Google, limiting the search to your site, find the top page for a city/place, scrape that page; and do all that completely automated.
    – pbhj
    Jun 26, 2019 at 20:52
  • They won't pay me because "I have ruined security of the page" by making URLs like that. Go figure. In the old version, slug was not verified so they added random letter to the end. I have removed random letter but I am checking complete slug so you can't just increase id and slug is variable length, so almost impossible to guess it...
    – kovinet
    Jun 26, 2019 at 21:12
  • 1
    Neither scheme seems like it would work. Either sue them for the money or do the work; it sounds like they're making things up to avoid paying.
    – pbhj
    Jun 26, 2019 at 21:40

1 Answer 1

3

Changing a URL by adding a character or two does not improve any form of security.

What are the client's goals for this site? Should it be publicly available to human visitors? Should it be only available to certain human visitors? Or should it be completely private, only for their own internal use?

If the website should be publicly available to human visitors, then "security" as in not being able to scrape the content is an impossible goal. Web browsers work by saving all the content (text, images, etc.) of a URL to the visitor's computer, so they will always be getting a copy. If you want people to be able to find a site and view it, then it needs to be intentionally open and accessible to all - humans and bots alike. Adding a couple of characters to the URL doesn't prevent bots from scraping, nor human visitors from copying and pasting.

If the website should only be available to certain human visitors, you can require logins or simple .htaccess passwords. Also be sure to use noindex tags for web crawlers that honor them.

If the website should be completely private, have them host it on their own computer. Help them set up a localhost and never push it to the web.

Regardless of the case, if you didn't have them sign a contract that explicitly defined requirements and payment schedule, don't do further work for them. It's your call whether to try to sue in small claims court to get them to pay. If you want to go that route, be sure to have copies of all communications. Also in the future, take the time to write up a requirements and payment schedule contract to protect you from this type of demand.

If you do have a contract, it will be easier to enforce by informing them that the requested work is complete and they must pay by a certain date or else you will be forced to take the matter to court.

In either case, you can also back up a copy of the site and remove it from their servers, and notify them that when payment is complete the site will be made live. Unfortunately, some clients take advantage and this is a good time to evaluate what you can do to protect yourself in the future.

1
  • Client's goals are to have a public site, completely indexed by google and other search engines, but at the same time make it hard for someone to write a simple bot and copy contents from the website. I know perfectly well that this is not possible, but they won't take my word for it, so I made the issue public with this question. Their reasoning is, that because of this "mistake" I made they won't pay me anything for 3 months worth of work, because I caused damage to them because their content was stolen by not adding extra random character to URL.
    – kovinet
    Jun 29, 2019 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.