Thursday 21 August 2014

Failing ssh connections on Centos 7 docker host

When I tried to connect to a container running on Centos 7 host running ssh service I was getting:

Read from socket failed: Connection reset by peer

I was running a container with ssh service inside it on Ubuntu using Dockerfile from: https://docs.docker.com/examples/running_ssh_service/

# sshd
#
# VERSION               0.0.1

FROM     ubuntu:12.04
MAINTAINER Thatcher R. Peskens "thatcher@dotcloud.com"

# make sure the package repository is up to date
RUN apt-get update

RUN apt-get install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:screencast' |chpasswd

EXPOSE 22
CMD    ["/usr/sbin/sshd", "-D"]

It worked perfectly on Ubuntu 14.04 as the docker host, but when I tried the same docker file build on Centos 7 host I was unable to connect to the ssh service running inside container. I've tried all the available solutions mentioned on the internet like disabling selinux, disabling firewalld , changing container ssh configuration from UsePAM yes to UsePAM no and generating ssh keys inside the container before running sshd daemon. Also used the docker file from here: https://github.com/tutumcloud/tutum-entos/blob/master/Dockerfile


Running netcat and using telnet to connect works.
But the ssh problem still persisted and after searching further found this discussion:
https://github.com/sameersbn/docker-gitlab/issues/122
which helped me resolve the problem. So basically to summarize the solution
set UsePrivilegeSeparation no in the sshd_config of the container and you should be able to login 
Dockerfile for the same adapted from (https://github.com/tutumcloud/tutum-centos) :

One nice tip learned from this troubleshooting is that you can enter a container using tool called nsenter as follows:

PID=$(docker inspect --format '{{.State.Pid}}' my_container_id)

nsenter --target $PID --mount --uts --ipc --net --pid

But got another problem, the session hangs right after logging in. working on finding a fix for that issue.
References:
https://github.com/sameersbn/docker-gitlab/issues/122
https://github.com/docker/docker/issues/6103
http://jpetazzo.github.io/2014/03/23/lxc-attach-nsinit-nsenter-docker-0-9/