Pelican blog with Fabric and FTP deployment part 2/2

FTP Deployment using ftputil for Pelican Blog.

Process of creating Fabric Deployment with FTP included:

1. Using Quickstarter for creating default pelican blog example.
2. Setting-up FTP Server for testing.
3. Adding ftp_upload task at Fabric default fabfile.py.
4. Exploring and development of ftp_upload task using ftputil and netrc as I've briefly predicted in post on 'Pelican blog with Fabric and FTP part1'

Development of FTP Fabric deployment

I've firstly tried to fight-out problem with ftp-server, but hence time limits, I had to bury investigating ftp-server itself (which meant to be a testbed only, not actual post).

Finally when I've found solution, I've researched how other's solutions looks like, but non of them were satisfying my needs - check stackoverflow link

I almost copy-pasted last solution, but made my tweaks to it that actually extends this with not only source but also target directory.

Finally solution looks like this:

def ftp_upload():
    netrc_data = netrc.netrc(env.netrc_file)
    authenticator = netrc_data.authenticators(env.ftp_host)

    def upload_dir(source, target):
        ftp_host.chdir(target)
        for dir_name, _, dir_fiels in os.walk(source):

            local = os.path.join(os.curdir, dir_name)
            local_strip = local.strip("./")
            local_for_remote = local_strip.strip(source)
            if not ftp_host.path.exists(target + local_for_remote):
                ftp_host.makedirs(target + local_for_remote)

            for file_ in os.listdir(local_strip):
                source_upload = local + "/" + file_
                target_upload = target + local_for_remote +"/" + file_
                if not os.path.isdir(source_upload):
                    logging.debug("uploads {},  {}".format(source_upload, target_upload))
                    ftp_host.upload(source_upload, target_upload)

    with ftputil.FTPHost(env.ftp_host, authenticator[0], authenticator[2]) as ftp_host:
        upload_dir(env.deploy_path, env.ftp_target_dir)

This solution uses netrc file as you see at the second line.

An example of netrc file that KEEP-IN-MIND needs to have a chmod 0600 .netrc mode, looks like this:

machine ftp.freebsd.org
    login anonymous
    password edwin@mavetju.org

You may use this solution with fabric as described below:

fab --set ftp_host=IP_ADDRESS_OR_DNS_NAME_OF_FTP,ftp_target_dir=/DIRECTORY_AT_FTP_SERVER_TO_WHICH_FILES_WILL_BE_UPLOADED/,netrc_file=/YOUR_ABSOLUTE_PATH_LOCAL_NETRC_FILE ftp_upload

i.e.

fab --set ftp_host=127.0.0.1,ftp_target_dir=/var/www/,netrc_file=/root/.netrc ftp_upload

Code commits done for this post:

Adds simplest version of fab-ftp using lftp

Adds ftp_upload task

Tools and applications used:

  • Docker (for ftp-server and for pelican-deployment tests)

  • Vi/VIM as my main editor

  • Tmux

LL (Lessons Learned)

1. Using docker and ftp service introduce networking problem.

Using ftp service called "pureftpd" as docker container was not a perfect solution as I've found out in a painfull manner.

I've tried to test my ftp_upload fabric task, but I could not even connect with simple ftp on second docker image nor via host itself.

I've stumbled upon error: 

500 I won't open a connection to 172.17.0.155 (only to 172.17.42.1)
ftp: bind: Address already in use

Solution:

I needed to change network settings of docker container to share all network with host - with adding at docker run parameter --net=host

2. Using pureftpd as testing ftp-service introduce too-much-connections-per-user error.

I was testing putting files to ftp-server, and I could not use ftputil nor lftp command to put files. I could only use simplest ftp command to put files to server.

Error I've got:

Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 756, in main
        *args, **kwargs
    File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 426, in execute
        results['<local-only>'] = task.run(*args, **new_kwargs)
    File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 173, in run
        return self.wrapped(*args, **kwargs)
    File "/app/fabfile.py", line 120, in ftp_upload
        upload_dir(env.deploy_path, env.ftp_target_dir)
    File "/app/fabfile.py", line 117, in upload_dir
        ftp_host.upload(source_upload, target_upload)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 475, in upload
        conditional=False, callback=callback)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/file_transfer.py", line 184, in copy_file
        target_fobj = target_file.fobj()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/file_transfer.py", line 94, in fobj
        return self._host.open(self.name, self.mode)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 205, in open
        host = self._copy()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 144, in _copy
        return self.__class__(*self._args, **self._kwargs)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 71, in __init__
        self._session = self._make_session()
    File "/usr/local/lib/python2.7/dist-packages/ftputil/host.py", line 131, in _make_session
        session = factory(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/ftputil/error.py", line 134, in __exit__
        raise TemporaryError(*exc_value.args, original_exception=exc_value)
ftputil.error.TemporaryError: 421 I can't accept more than 1 connections as the same user
Debugging info: ftputil 3.3.1, Python 2.7.6 (linux2)

Solution:

I tried to change configuration, set PerUserLimit's and other, but it didn't work.

Finally in my desperation and a little bit too much time spend on tracking this issue then actual development I've dropped this thing for now and changed ftp server to much less "secure" - to proftpd with predefined docker image on x64 bit - called hauptmedia/proftpd - I recommend you to check his github account projects :).

This container works properly and for now is my testbed.

When I've got more time I'll investigate problem with pureftpd and it's "security-by-obscurity" so-to-speak problem.

Acknowledgements

FTPUtil Documentation

Simple FTP Mirror - By Thimo Kraemer

What's next

1. Using pelican-bootstrap3 theme (or other), extend possibilities with adding fancybox as image viewer.
2. See ISSSUES task-list for this 'mini' growing project :)

See you at next Wednesday :)

Comments

comments powered by Disqus