A good firewall is rarely a bad thing, yet it can make some actions more complicated. Because some hosters only whitelist certain IPs, sometimes you might need to import a shop through a proxy. This article shows two methods of using the hypernode-importer for automatically migrating or copying Magento shops to another server in combination with a proxy server.

Note: this is advanced usage of the hypernode-importer, if you are just looking to migrate your shop to Hypernode, check out this article.

The use-case for this article is as follows: you have two Hypernodes, or one Hypernode and another server and only one has access to a remote SSH server. This article explains how to import a shop to a Hypernode, through a second server from a third server.

When the Hypernode can not directly access the third server it can not perform the import. The second server can however connect to the third server. In this scenario we do not want to import to the second server but to the Hypernode instead.

Method 1. Using a ProxyCommand

You can configure SSH to use a ProxyCommand for specific hosts. There are two ways you can configure this. There is the classic netcat method and the more recent stdio forwarding feature built in to SSH.

The netcat method does not support having a different login user for the proxy and the destination. It also sometimes leaves lingering nc processes behind that won’t get cleaned up until you kill them. This article explains both for completeness but we recommend that if you need to resort to proxying your SSH connections you use the ssh -W method and not the netcat method.

• Create a SSH config file

mkdir -p ~/.ssh
touch ~/.ssh/config
chmod 0600 ~/.ssh/config

• Edit the config file so it contains the following directives

If you have three servers, hypernode1.hypernode.io, hypernode2.hypernode.io, hypernode3.hypernode.io and you want to import from hypernode3 to hypernode1 but by proxying through hypernode2, you can use this SSH configuration on the hypernode1 server.

Host * !hypernode2.hypernode.io
    ProxyCommand ssh -W %h:%p app@hypernode2.hypernode.io

The Host specifies that this rule should hold true for all hosts (the wildcard *) except for the hypernode2.hypernode.io host. This negation is needed because otherwise the tunnel will try to tunnel to itself creating an infinite loop. This would consume memory until the process is OOMkilled and the user gets kicked out of the shell.

• Using netcat instead (optional)

If you do not want to use the builtin ssh -W (if it does not work for some reason) you can always this the alternative configuration using netcat.

Host * !hypernode2.hypernode.io
    ProxyCommand ssh hypernode2.hypernode.io nc %h:%p  2> /dev/null

This is effectively the same thing but with the limitations as mentioned above.

• Test if you can make the connection from from the first to the second server

You need to make sure you can connect to the proxy from the server you want to import to. It is best to have passwordless SSH login setup so you can login using a key-pair. To accomplish this you need to either manually add a public key to the ~/.ssh/authorized_keys file on the second machine or use the keymanager in our service panel. You can use SSH forwarding if you want to use the same identity to connect to the second server as you used to connect to the first server.

app@levivb-hypernode1-magweb-do:~$ ssh hypernode2.hypernode.io -oPasswordAuthentication=no echo 'can connect to $HOSTNAME'
can connect to er59on-hypernode2-magweb-do.nodes.hypernode.io

• Test if you can make the connection from the first to the third server through the second server

app@levivb-hypernode1-magweb-do:~$ ssh hypernode3.hypernode.io -oPasswordAuthentication=no echo 'can connect to $HOSTNAME'
can connect to 5vi9fo-hypernod3-magweb-do.nodes.hypernode.io

• Check if the connection actually uses the tunnel

app@levivb-hypernode1-magweb-do:~$ ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20 &
app@levivb-hypernode1-magweb-do:~$ ps aux | grep hypernode3
app      17200  0.0  0.2  41528  2852 pts/8    T    08:43   0:00 ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20
app      17203  0.0  0.2  43608  2992 pts/8    T    08:43   0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io

Note that there is an SSH process that uses the configured ProxyCommand to connect to the second server. This connection tunnels your connection to the third server.

• Performing the import using the bouncer

The hypernode-importer will try to parse the ProxyCommand from the SSH config and use that in all steps requiring a remote SSH connection. This means you do not need to add any special flags to the importer to import through the tunnel once the SSH configuration is in place.

app@levivb-hypernode1-magweb-do:~$ hypernode-importer --host hypernode3.hypernode.io --verbose --dry-run

If you look in another shell to the running processes once again using ps aux | grep ssh you will see that the hypernode-importer is using the specified proxy to import the shop.

app@levivb-hypernode1-magweb-do:~$ ps aux | grep ssh
app      17200  0.0  0.2  41528  2852 pts/8    T    08:43   0:00 ssh hypernode3.hypernode.io -oPasswordAuthentication=no sleep 20
app      17203  0.0  0.2  43608  2992 pts/8    T    08:43   0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io
app      18529  0.0  0.2  43608  2996 pts/8    S+   08:57   0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io
app      18540  1.0  0.2  41524  2832 pts/8    S+   08:57   0:00 ssh -4 -p 22 app@hypernode3.hypernode.io -L 34082:mysqlmaster.hypernode3.hypernode.io:3306 -N
app      18541  1.0  0.2  43608  2992 pts/8    S+   08:57   0:00 ssh -W hypernode3.hypernode.io:22 app@hypernode2.hypernode.io

• Clean up the .ssh/config file

If you do not need the proxy anymore (after you’ve finished importing) it is good practice to clean up the configured ProxyCommand so in the future you don’t accidentally proxy all outgoing SSH connections through the configured tunnel.

Method 2. Using SSH forwarding

The hypernode-importer has many strategies for dealing with all kinds of source servers. The Hypernode platform is homogeneous in its configuration, but the servers at various different hosting providers that people want to move their shop from to our platform have many different operating systems, settings and configurations. We try to make the tool as plug-and-play as possible but it does sometimes happen that our automation can not deal with whatever it encounters on the remote server.

Using a ProxyCommand limits the amount of strategies the importer can use. If the previous method was unsuccessful in your specific situation, there is something you can do to try and make the importing environment more friendly to the hypernode-importer.

• Forwarding the remote SSH port on the bouncer

If you do not mind exposing the SSH port (that is presumably firewalled to only give access to the second server) to the world, you can try to forward that remote SSH port on the third server on the second server. It works like this:

1. Hypernode 2 can connect to Hypernode 3
2. Hypernode 1 can not connect to Hypernode 3
3. Hypernode 1 can connect to Hypernode 2
4. Hypernode 2 can forward the SSH port of Hypernode 3 by binding it to a port on itself
5. Hypernode 1 can then SSH to Hypernode 2 on that bound port and it will connect transparently to Hypernode 3

Note: only do this if you know what you are doing. If the SSH port on the third server is firewalled and not accessible to the outside world that is probably for a reason. The maintainers of the third server might not appreciate you exposing their firewalled port. It might also introduce a security risk.

This command forwards the remote SSH port 22 of the third server to the unprivileged port 49510 on the second server.

app@er59on-hypernode2-magweb-do:~$ ssh -L 49510:hypernode3.hypernode.io:22 -g hypernode3.hypernode.io

The -L specifies that connections to the given TCP port are to be forwarded to the given host and port on the remote side. The -g allows remote hosts to connect to the forwarded ports (warning: potentially dangerous).

• Import from the second server while actually transparently importing from the third server

You can now run the importer on the first server specifying to import from the second server using the forwarded port. This will import the shop from the third server to the first server while bouncing through the second server.

This sometimes has a higher success rate than the ProxyCommand approach because there are less moving parts. The hypernode-importer thinks it is just importing directly from the second server (while it is actually connecting to the third).

app@levivb-hypernode1-magweb-do:~$ hypernode-importer --host hypernode2.hypernode.io --port 49510 --verbose --dry-run

• Clean up the forwarded tunnel

After you’ve finished importing be sure to clean up the port forwarding connection on the second server.

app@er59on-hypernode2-magweb-do:~$ ps aux | grep [s]sh | grep '\-L' | awk '{print $2}' | xargs kill