A good firewall is rarely a bad thing, yet it can make some actions more complicated. Because some hosters only whitelist certain IPs, sometimes you might need to import a shop through a proxy. This article shows two methods of using the hypernode-importer
for automatically migrating or copying Magento shops to another server in combination with a proxy server.
Note: this is advanced usage of the hypernode-importer
, if you are just looking to migrate your shop to Hypernode, check out this article.
The use-case for this article is as follows: you have two Hypernodes, or one Hypernode and another server and only one has access to a remote SSH server. This article explains how to import a shop to a Hypernode, through a second server from a third server.
When the Hypernode can not directly access the third server it can not perform the import. The second server can however connect to the third server. In this scenario we do not want to import to the second server but to the Hypernode instead.
Method 1. Using a ProxyCommand
You can configure SSH to use a ProxyCommand for specific hosts. There are two ways you can configure this. There is the classic netcat method and the more recent stdio forwarding feature built in to SSH.
The netcat
method does not support having a different login user for the proxy and the destination. It also sometimes leaves lingering nc
processes behind that won’t get cleaned up until you kill them. This article explains both for completeness but we recommend that if you need to resort to proxying your SSH connections you use the ssh -W
method and not the netcat method.
- Create a SSH config file
1 2 3 |
mkdir -p ~/.ssh touch ~/.ssh/config chmod 0600 ~/.ssh/config |
- Edit the config file so it contains the following directives
If you have three servers, hypernode1.hypernode.io
, hypernode2.hypernode.io
, hypernode3.hypernode.io
and you want to import from hypernode3
to hypernode1
but by proxying through hypernode2
, you can use this SSH configuration on the hypernode1
server.
1 2 |
Host * !hypernode2.hypernode.io ProxyCommand ssh -W %h:%p app@hypernode2.hypernode.io |
The Host
specifies that this rule should hold true for all hosts (the wildcard *
) except for the hypernode2.hypernode.io
host. This negation is needed because otherwise the tunnel will try to tunnel to itself creating an infinite loop. This would consume memory until the process is OOMkilled and the user gets kicked out of the shell.
- Using netcat instead (optional)
If you do not want to use the builtin ssh -W
(if it does not work for some reason) you can always this the alternative configuration using netcat
.
1 2 |
Host * !hypernode2.hypernode.io ProxyCommand ssh hypernode2.hypernode.io nc %h:%p 2> /dev/null |
This is effectively the same thing but with the limitations as mentioned above.
- Test if you can make the connection from from the first to the second server
You need to make sure you can connect to the proxy from the server you want to import to. It is best to have passwordless SSH login setup so you can login using a key-pair. To accomplish this you need to either manually add a public key to the ~/.ssh/authorized_keys
file on the second machine or use the keymanager in our service panel. You can use SSH forwarding if you want to use the same identity to connect to the second server as you used to connect to the first server.
1 2 |
app@levivb-hypernode1-magweb-do:~$ ssh hypernode2.hypernode.io -oPasswordAuthentication=no echo 'can connect to $HOSTNAME' can connect to er59on-hypernode2-magweb-do.nodes.hypernode.io |
- Test if you can make the connection from the first to the third server through the second server
1 2 |
app@levivb-hypernode1-magweb-do:~$ ssh hypernode3.hypernode.io -oPasswordAuthentication=no echo 'can connect to $HOSTNAME' can connect to 5vi9fo-hypernod3-magweb-do.nodes.hypernode.io |
- Check if the connection actually uses the tunnel
1 2 3 4 |
app@levivb-hypernode1-magweb-do:~$ ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20 & app@levivb-hypernode1-magweb-do:~$ ps aux | grep hypernode3 app 17200 0.0 0.2 41528 2852 pts/8 T 08:43 0:00 ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20 app 17203 0.0 0.2 43608 2992 pts/8 T 08:43 0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io |
Note that there is an SSH process that uses the configured ProxyCommand to connect to the second server. This connection tunnels your connection to the third server.
- Performing the import using the bouncer
The hypernode-importer
will try to parse the ProxyCommand
from the SSH config and use that in all steps requiring a remote SSH connection. This means you do not need to add any special flags to the importer to import through the tunnel once the SSH configuration is in place.
1 |
app@levivb-hypernode1-magweb-do:~$ hypernode-importer --host hypernode3.hypernode.io --verbose --dry-run |
If you look in another shell to the running processes once again using ps aux | grep ssh
you will see that the hypernode-importer
is using the specified proxy to import the shop.
1 2 3 4 5 6 |
app@levivb-hypernode1-magweb-do:~$ ps aux | grep ssh app 17200 0.0 0.2 41528 2852 pts/8 T 08:43 0:00 ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20 app 17203 0.0 0.2 43608 2992 pts/8 T 08:43 0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io app 18529 0.0 0.2 43608 2996 pts/8 S+ 08:57 0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io app 18540 1.0 0.2 41524 2832 pts/8 S+ 08:57 0:00 ssh -4 -p 22 app@hypernode3.hypernode.io -L 34082:mysqlmaster.hypernode3.hypernode.io:3306 -N app 18541 1.0 0.2 43608 2992 pts/8 S+ 08:57 0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io |
- Clean up the
.ssh/config
file
If you do not need the proxy anymore (after you’ve finished importing) it is good practice to clean up the configured ProxyCommand
so in the future you don’t accidentally proxy all outgoing SSH connections through the configured tunnel.
Method 2. Using SSH forwarding
The hypernode-importer
has many strategies for dealing with all kinds of source servers. The Hypernode platform is homogeneous in its configuration, but the servers at various different hosting providers that people want to move their shop from to our platform have many different operating systems, settings and configurations. We try to make the tool as plug-and-play as possible but it does sometimes happen that our automation can not deal with whatever it encounters on the remote server.
Using a ProxyCommand
limits the amount of strategies the importer can use. If the previous method was unsuccessful in your specific situation, there is something you can do to try and make the importing environment more friendly to the hypernode-importer
.
- Forwarding the remote SSH port on the bouncer
If you do not mind exposing the SSH port (that is presumably firewalled to only give access to the second server) to the world, you can try to forward that remote SSH port on the third server on the second server. It works like this:
- Hypernode 2 can connect to Hypernode 3
- Hypernode 1 can not connect to Hypernode 3
- Hypernode 1 can connect to Hypernode 2
- Hypernode 2 can forward the SSH port of Hypernode 3 by binding it to a port on itself
- Hypernode 1 can then SSH to Hypernode 2 on that bound port and it will connect transparently to Hypernode 3
Note: only do this if you know what you are doing. If the SSH port on the third server is firewalled and not accessible to the outside world that is probably for a reason. The maintainers of the third server might not appreciate you exposing their firewalled port. It might also introduce a security risk.
This command forwards the remote SSH port 22 of the third server to the unprivileged port 49510 on the second server.
1 |
app@er59on-hypernode2-magweb-do:~$ ssh -L 49510:hypernode3.hypernode.io:22 -g hypernode3.hypernode.io |
The -L
specifies that connections to the given TCP port are to be forwarded to the given host and port on the remote side. The -g
allows remote hosts to connect to the forwarded ports (warning: potentially dangerous).
- Import from the second server while actually transparently importing from the third server
You can now run the importer on the first server specifying to import from the second server using the forwarded port. This will import the shop from the third server to the first server while bouncing through the second server.
This sometimes has a higher success rate than the ProxyCommand
approach because there are less moving parts. The hypernode-importer
thinks it is just importing directly from the second server (while it is actually connecting to the third).
1 |
app@levivb-hypernode1-magweb-do:~$ hypernode-importer --host hypernode2.hypernode.io --port 49510 --verbose --dry-run |
- Clean up the forwarded tunnel
After you’ve finished importing be sure to clean up the port forwarding connection on the second server.
1 |
app@er59on-hypernode2-magweb-do:~$ ps aux | grep [s]sh | grep '\-L' | awk '{print $2}' | xargs kill |