Today's Question:  What does your personal desk look like?        GIVE A SHOUT

Ruby net-scp cannot scp multiple files with asterisk(*)

  Pi Ke        2016-10-20 03:00:13       4,987        7    

net-ssh/net-scp is a Ruby gem which can be used to scp files between different *nix machines. It's similar to how the *nix scp command. It can be used to scp a file or a directory. However, it seems it has some problem to scp multiple files using pattern *.

For example, below script is supposed to download all files from remote directory to local directory:

require 'net/scp'

host        = 'testmachine'
login       = 'testaccount'
password    = "testpassword"
remote_path = '/tmp/remote_dir/*'

Net::SCP.start(host, login, :password => password) do |scp|
  scp.download(remote_path, '.')
end

The execution result of this script will produce some SCP error.

/home/kepi/.rvm/gems/ruby-2.2.2/gems/net-scp-1.2.1/lib/net/scp.rb:366:in `block (3 levels) in start_command': SCP did not finish successfully (1): scp: /tmp/remote_dir/*: No such file or directory (Net::SCP::Error)

It says that no file can be found, but actually there are files present. 

The root cause of this issue is actually being that * is escaped when the command is being built. Basically it's in the start_command method. When shellescape method is being called, * will be escaped. See below code in scp.rb.

str.gsub!(/([^A-Za-z0-9_\-.,:\/@\n])/n, "\\\\\\1")

Since * is escaped, the actual command being executed would look like

scp -f /tmp/remote_dir/\\*

This command will fail. Actually the issue should be resolved by updating the shellescape method to not escape *.

str.gsub!(/([^A-Za-z0-9_\-.,:\/@\n\*])/n, "\\\\\\1")

But before this issue can be actually fixed in the official gem, there are a few workarounds for this issue. The first one is to define your own shellescape method for the path which contains asterisk. Since the shellescape in scp.rb looks like

# Imported from ruby 1.9.2 shellwords.rb
def shellescape(path)
  # Convert path to a string if it isn't already one.
  str = path.to_s

  # ruby 1.8.7+ implements String#shellescape
  return str.shellescape if str.respond_to? :shellescape

  # An empty argument will be skipped, so return empty quotes.
  return "''" if str.empty?

  str = str.dup

  # Process as a single byte sequence because not all shell
  # implementations are multibyte aware.
  str.gsub!(/([^A-Za-z0-9_\-.,:\/@\n])/n, "\\\\\\1")

  # A LF cannot be escaped with a backslash because a backslash + LF
  # combo is regarded as line continuation and simply ignored.
  str.gsub!(/\n/, "'\n'")

  return str
end

Above code shows that if a str has shellescape defined, that method will be called and returned. So in your own script, you can do something

require 'net/scp'

host        = 'testmachine'
login       = 'testaccount'
password    = "testpassword"
remote_path = "/tmp/remote_dir/*"

def remote_path.shellescape
  p "In self defined method " + self.to_s
  self
end

Net::SCP.start(host, login, :password => password) do |scp|
  scp.download(remote_path, '.')
end

With this, multiple files can be downloaded.

Another alternative you can do is to download the whole directory if you want to download all files in a directory. The code would look like.

scp.download! "#{source_dir}", "#{target_dir}", :recursive => true

This method will create a directory in target_dir and download all files from source_dir to the newly created dir.

RUBY  NET-SCP  ASTERISK  MULTIPLE FILES 

Share on Facebook  Share on Twitter  Share on Weibo  Share on Reddit 

  RELATED


  7 COMMENTS


Anonymous [Reply]@ 2017-08-31 20:44:04

It didn't work for me

Ke Pi [Reply]@ 2017-09-01 05:14:06

Both methods are not working for you? Have you checked the shellescape method in scp.rb of your version of Ruby? Maybe the code has been changed there.

Anonymous [Reply]@ 2017-09-01 16:43:09

I use ruby 2.1.6, and remote_path string after running the shell escape method is :: " /tmp/\* ". It returns at line 7 (return str.shellescape if str.respond_to? :shellescape)

But I still get this error :: :in `block (3 levels) in start_command': SCP did not finish successfully (1): scp: /data/ic/logs/tomcat/*: No such file or directory (Net::SCP::Error)

 

[root@active]# ruby --version

ruby 2.1.6p336 (2015-04-13 revision 50298) [x86_64-linux]

[root@active]#

 

 

Ke Pi [Reply]@ 2017-09-02 03:43:06

Are you sure /data/ic/logs/tomcat/ is existing?

Can you send the complete code you are running?

Anonymous [Reply]@ 2017-09-01 17:36:12

How should be the remote_string be ? 

 

As you know the start command, irrespective of the string given in the code I'm over-righting it as

 

def start_command(mode, local, remote, options={}, &callback)

        session.open_channel do |channel|

 

          if options[:shell]

            escaped_file = shellescape(remote).gsub(/'/) { |m| "'\\''" }

            command = "#{options[:shell]} -c '#{scp_command(mode, options)} #{escaped_file}'"

          else

            command = "#{scp_command(mode, options)} #{shellescape remote}"

            puts "#{command}"

            command = "scp -f -r /tmp/*.log"

          end 

 

Is this right /tmp/*.log ?

Ke Pi [Reply]@ 2017-09-02 03:44:28

Should be working though not recommended.

Anonymous [Reply]@ 2017-09-11 12:40:09

Yeah I know its not recommended, I'm just playing with the ruby script, so trying to override the command line argument and trying to work with hardcoded string. Still not working. 

 

Anyways, I read some document that says scp folder is possible by setting the flag recursive to true and single file copy is also possible. The doc also says that the files with wild characters are not possible to copy remotely. 

So I modified my script to get the ls of the path and copying it files by file.