Looking for a 'crawler friendly' dedicated server company

Hi there

We have created a crawler/spider which is intended to visit the homepage (only) of all the domain names that we can find - currently approx 250m of them. We are running into trouble because a lot of domain names don't resolve and our current dedicated server provider keeps sending us automated alerts regarding our activity - despite trying to talk to them about what we are doing, we keep receiving the same.

Our plan is when we find a domain that does not resolve that we put it on a list to only try it infrequently (maybe once a month) to see if it becomes "live". However we are not able to do even a full crawl without having a problem

So we have two things:

1) Can anyone recommend a company that is happy to have someone on their network doing this? We aren't overloading servers, are being 'polite' including looking at robots.txt and are only looking at the homepage.

2) If anyone has any suggestions for how we can reduce these issues, it would be welcome. This is the first time that we have created such a broad crawler and therefore anyone with knowledge like this would be really appreciated

Thanks


Similar Content



Hetzner DNS console: only manually add domain names?

I recently got a server (AX41-NVMe) with Plesk at Hetzner. Via the DNS console I added the base domain name, where I also want to reach the server, and that works fine.

What I'm running into now is Hetzner telling me to manually add all domain names to the DNS console in order to use them with the server. But that's crazy, isn't it? There are hosting companies that also rent the servers and they cannot manually add every domain name in the DNS console, I think.

I'm not used to it that way either. The server I rent from Leaseweb allows me to simply add a domain name to cPanel and it will be automatically recognized by the worldwide DNS servers.

For me the Hetzner DNS console is not a workable solution in this way. So I thought to solve it with the Plesk extension for Digital Ocean. Works great! But unfortunately no more than 50 domain names are allowed to be added to their DNS. I want to add 100 domains, so this solution doesn't work either.

So my question to you is:

How can I make my domain names reachable, without manually adding them all to Hetzner's DNS console?

Fastest way to set up a domain on a dedicated server, using Cloudflare nameservers?

Hello, folks...

I'm looking to have a live website at some-domain.com, hosted on a dedicated server, while using CloudFlare nameservers. And I need to find the fastest way to do this from scratch.

I'm getting a bit confused with all the steps required. I also think I'm waiting unnecessarily for things to happen. This shouldn't be taking so long.

My current understanding of the process is this:

1) Register the domain with a domain registrar.

2) Login to my hosting account's cPanel and add the domain as an addon domain.

3) Add a DNS zone for the domain in my dedicated server's control panel (not cPanel).

4) Login to the domain registrar. Add custom nameservers to the domain, and point them to my server's nameservers.

5) Wait 2 days for the nameservers to propagate.

6) Login to CloudFlare, add a new site and let Cloudflare import the DNS zones.

7) Login to the domain registrar and change the nameservers of the domain to those Cloudflare gives me.

8) Wait 2-3 days for the nameservers to propagate.

9) Test the site.
Can you please help me streamline this process?

Thanks!

Do I need a Dedicated Server?

I have been hosting my sites on my current hosting company for 7 years. TBH, I am happy with them most of the time.

Since they were sold to another company, the tech support is getting worse. Each time it takes a long time for them to figure out what's going on. They always said the problem is on my side. But it turns out all the issues I complained locate on the server side. I had enough with those offshore tech support.

The second thing is: on my current plan, the server blocked a lot of proxy/Tor access, maybe for anti-spamming or whatever. And host-tacker shows some "HostNotFound" results, which I have not seen it from my friend's sites (we both run the same type of sites). So I lost some traffic as well.

That's why I am thinking to move my sites out.

My site is not big. Currently, I have 2500-3000 unique visits/day, and bandwidth/m is about 700 GB. I am thinking to try a dedicated server plan this time. Then I will have the faster page loading speed, which means better SEO, I guess? Hopefully I can get more traffic and more visits. And I won't get bothered by bad neighbors any more.

I checked a few dedicated plans, can you tell me what these mean? esp WAN uplink and LAN Uplink? Should it be included in total bandwidth?

1,000 Megabit WAN Uplink
1,000 Megabit LAN Uplink
25,000 GB Included Transfer

I read some articles about Dedicated IP vs Shared IP. It seems Dedicated IP is better than Shared IP. But for a small site like mine, do I really need a dedicated server?

I am thinking of mojohost as some friends recommend them. Do they have a coupon or I can negotiate with them for a discount?

Any advises are appreciated.

Migrating clients websites from VPS to Dedicated server with no downtime

Hi,

I have 50+ Cpanel accounts on my VPS, I'm looking to migrate to a dedicated server from another provider,
I want to know if there's a way to migrate from the VPS to the new server with no downtime?
I have access to most of my client's domain names, also most of my clients are using Cloudflare (My account).
Can anyone advise me on what's the best approach here?

Thank you.

medium price managed dedicated server wthh cpanel for online auction

I am hosting 2 domain in a managed dedicated server. Recently experienced more downtime than usual. Among my 2 domain 1 is Online auction website with 2000 + members and always with live visitor 40 +..
Can you suggest me a web hosting company offer medium price managed server?
Thanks

How companies register IP addresses for their dedicated servers?

What is the path the companies take to have IPs registered to their own business names in dedicated server business? So basically if someone looks up an IP it shows a company name of whoever owns or leases the IPs.

VPN for an API connecting to a web application hosted in dedicated server?

We have a client running a web application on a dedicated server in Finland (Hetzner). Client is based in Asia. They're trying to integrate a third party API, but that provider says they can't connect to a Finland IP and need an Asian one. Hetzner doesn't have a presence in this Asian country, so I am wondering if a VPN could be used in some way to resolve this issue.

But, it's not clear how. Any suggestions would be much appreciated.

Thanks in advance.

Is Serveria.com a scam?

I came across the company serveria.com from seeing multiple adverts on this forum offering discounted dedciated servers, so I decided to order a server as they were in the EU and offered a fair server at a fair price.

Several days later I am still waiting for the server, the order has been paid in full and is just 'Pending' so I opened a ticket which has yet to be answered. I have even sent an email but heard nothing. With a company offering dedciated servers on this forum and have a long history I'm not sure what to do, is there an admin to report this too? Has anyone had issues with this company? Are there no checks made on companies offering dedicated servers on this forum? Its put me off now looking for any other servers via this forum.

Welp, Guess I need a Managed Dedicated Server Provider...who do you recommend?

This current admin company is driving me insane! They are literally making my server worse...

I have lost so much money in production time of not being able to develop websites!

This sever is for basic business sites.

So I am moving this server from dedicated self-managed (with 3rd party admin) and now need a fully managed server.

This is just to host multiple websites. so we very rarely have issues on the server.

I need cpanel/whm

I am used to having multiple processors with multiple cores, but they are quite old, so maybe the more powerful processors can beat it?

It's an old server but has been a workhorse for development with no real issues.

i'd like the equivalent or better. Ideally under $200 month ($150 would be great!)

-- is this even possible?

who do you recommend? Any black friday/cyber Monday specials?



--------current server specs---
Intel 2x L5630
Dedicated Server
Operating System: cPanel/WHM (CentOS 7 x64)
Bandwidth: 20TB on 1Gbps Port
Service Title: Intel 2x L5630
Service Options: Service Plan: Intel 2x Xeon L5630 Westmere 4-Core Dell Node
Operating System: Linux- CentOS 64-bit with cPanel/WHM 64-bit
Hard Drive 1: 500GB HDD
Hard Drive 2: None (+$0.00)
Hard Drive 3: None (+$0.00)
Hard Drive 4: None (+$0.00)
Hard Drive 5: None (+$0.00)
Hard Drive 6: None (+$0.00)
Raid Card- LSI 9260-8i 6G w/ 512MB Cache: Included / No Raid (+$0.00)
Power Supply: Dual Power (+$0.00)
RAM: 24GB (+$0.00)
Bandwidth: 20.0TB on 1Gbps Port (+$0.00)
IPv4 Addresses: 5 Usable (/29) (+$0.00)

Is there such a thing as AFFORDABLE admin/management WITH Good customer service...or nah?

I have been trying to find server admins for a Cpanel dedicated machine.

I find my experience has been the following in the last few years.

1) CHEAP -- great if nothing is needed. Low skills, low service. Outsourced to 3rd world countries. Only really available to do work during flip-flop hours (I am in USA). Never worth it in the end.

2) Affordable with either

a) GREAT service and low skills (still outsourcing to cheap freelancers?)
or
b) Poor service and low skills

3) Individual, who's pricing changes, availability changes, or business communications are not fully professional.

Is there anyone out there that has GREAT SERVICE and DECENT Pricing? or do I just need to bite it and pay $200 a month (twice the cost of the server lease) for decent administrative services?

My most recent and present experience is with a provider here from WHT. They are in the EU. Their reps log in from Germany and Greece. Usually in the US daytime, work is not done.

They told me they were UNABLE to repair the Kernel. Somehow, a previous admin had installed kernalcare and they claim it was removed and now causing errors in csf/lfd. (they didn't know how to fix it)
My only solution was to reinstall os or buy a New server from them.

Let me add, they were very pushy on how "old" my machine was and that was the reason support was slow.

they had problems backing up the system and couldn't guarantee it would work right again unless I bought anew server from them.

I didn't, they are in the EU and I am in the US, so not worth the GDPR compliance hassles.

So Did the reinstall with my DC.

Guess what, same problems...and more. For days. Their "pro-active" admin didn't catch these errors. I had to report them. eventually many of them got them fixed.

Except now, most sites are NOT working properly.

Basic things like .htaccess file and default.htm not recognized and working.

They told me because there are so many possibilities of php/configurations, we need to go through every site and test every function, then let them know, site by site, issue by issue.

Except they all worked fine before...

I have lost a LOT OF TIME and MONEY with this.

So, now, once again, looking for a company that can work with CPANEL and not blame the last admin or the server on their inability to manage it.

Anyone else have this issue?

how did you resolve it, just keep trying and trying, month after month with company after company?

Normally, I would just go with a dedicated machine that came with management. I didn't have these problems when I did. But the servers cost 3 times as much.

I guess I am asking, am I missing something or Is this just part of saving money on a dedi? No way around it except to just pay for the topline?