Microlink Proxy:
How it works

June 7, 2021 (1 year ago)

All

plans come with a built-in proxy layer that does automatic proxy resolution and rotation based on the input URL.

What's wrong with URLs

The Internet is a wild place where every URL is different. That's an issue especially when you are doing things at scale.

metascraper issue

facing with fetching problems.

When you are surfing the net and visit a website, there are situations you can't handle at all:

The are geographic location restrictions (e.g., visiting a site from China).
Your IP address is blacklisted (e.g., performing a request from a data center).
You have to validate your identity (e.g., filling a CAPTCHA).

You can quickly face these problems when sending enough traffic to popular sites, like Instagram or Bloomberg, which, in turn, makes getting a successful response inconsistent.

Resolving URLs gracefully

Wouldn't it be nice if every time you want to retrieve data from any URL, the Microlink API just returned the data all while handling any of the problems above transparently?

No additional setup needed, just a Microlink Pro plan.

That's exactly what Microlink Proxy layer does. No failures. No maintenance. You don't need to do anything additional. It just works™.

The Microlink Proxy is enabled for all Microlink Pro plans and it will handle all the situations for the

Top 500

most popular worldwide websites, in a transparent way, by default.

Just for informational purposes, you can check if a specific request was resolved using the proxy layer by checking x-fetch-mode response header.


$ microlink https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance

 SUCCESS  3.64 kB in 329ms

 cache HIT (19h 20m 54.8s left)
  mode prerender-proxy (8.5s)
   uri https://pro.microlink.io/?url=https://www.bloomberg.com/news/articles/2016-05-24/as-zenefits-stumbles-gusto-goes-head-on-by-selling-insurance&filter=statusCode
    id iad:paw98q36bHEIlAgsFBzy5

If the value there is prefixed by `proxy` then it was handled by the proxy layer.

As you can see, the proxy layer respects other query parameters, such as prerender or ttl.

Also, you can always provide your own proxy, being HTTP, HTTPS, or SOCKS (v4 and v5) protocols supported.

Last but not least, Microlink Proxy is a cross-feature: it's available for all the Microlink products.

Join the community

All of these improvements or features are community driven: We listen to your feedback and act accordingly.

Whether you are are building a product and you need fancy previews, you’re an indie hacker or simply you like frontend stuff, come

chat

with us 🙂.