Scripting zmprov with Ruby (getting stdout/stderr)

[Chewie71]

This is going to be long....just a warning.

First let me tell you how I got started on this. I'm working on some scripts that will autosubscribe users to calendars and folders. This will eventually be for 15000+ accounts. I originally started off by running a single zmprov command every time I wanted to perform some action, but that ends up being 3-5 zmprov commands per user and the open/run command/close time for each zmprov command was several seconds. OK for 10 users, not ok for 10,000. So I discovered that I could open a zmprov prompt once and then send it all the commands I wanted (sm, gaf, cf, etc...) for as many users as I wanted....and it would be super fast as I didn't have to keep opening and closing the zmprov prompt. So that's the background....here's the problem.

I'm writing my scripts in Ruby. It opens the zmprov prompt and then sends a series of commands (sm, gaf, cf, etc...), checks for output at each command, does something based on the output, and then move on to the next command. At least that's what I want it to do...

This is all well and good as long as I only check stdout. If I want to check both stdout and stderr....I start running into problems with blocking. It seems that zmprov is not releasing the pipe?/stream? on stdout/stderr so that the script gets control back...and so it sits and waits forever if I check stderr and no error was returned (i.e. the command worked)...or if I get an error then stdout is empty and blocks.

I've found and tried several different ways... IO.readpartial, threads (1 for each of stdin/stdout/stderr) with Open3.popen3, Open4.popen4(which has some threading built in). So far I've not found a single method that will let me do what I want...they all end up with blocking problems, or the output returned is not complete.

Apparently this is a difficult problem with zmprov not closing the stdout/stderr pipes until zmprov itself is closed.....so handling a ton of sm/gaf/cf/... output before closing zmprov for good is nigh impossible.

The only workaround I've come down to is to send my initial zmprov command like this 'zmprov 2>&1'... which will force both stdout/stderr to come through on stdout. Then I have to use some string matching to catch the errors manually by searching for the string 'ERROR:'. It's kind of kludgy and I don't normally like kludgy solutions if I can do it right in the first place....but it may be my only option since I can't seem to get it to work the right way. I have yet to try it though and prove that it works.

I'm just wondering if any ambitious scripters out there have figured a way around this so that they can send multiple commands through to zmprov and capture both stdout/stderr without blocking problems?

I can include code snippets if anyone is interested, but this is already pretty long.

Thanks,
Matt

[p24t]

I found some stuff relating to Open3, but I've never really used it before.

Module: Open3

require 'open3'
Open3.popen3('ls -lR bad_file_listing') {|stdin, stdout, stderr| puts stderr.read}

[Chewie71]

I'd figured out the Open3.popen3 stuff pretty well. The problem is in the stdout/stderr pipes. I don't know exactly how it all works, but if you write a program that does something like this...

stdin, stdout, stderr = Open3.popen3("zmprov")
stdin.puts("selectMailbox MYACCOUNT")
stdin.puts("getAllFolders")
stdin.puts("selectMailbox BADACCOUNT")

stdout.each do |line|
  puts line
end

stderr.each do |line|
  puts line
end

It opens the 'zmprov' command line and then sends the sm/gaf/sm commands through to zmprov. It all works (sort of) if all you ever want to read is from stdout....but if you want to read stderr too (like I send a bogus account name to that second sm)...you can't. Because of the fact that the program never closes the zmprov command....the stdout/stderr pipes never close and send an EOF (or whatever it's called) to the program so it never knows when to stop reading from the pipe and just sticks in an infinite loop.

Anyway...too much info probably....but I haven't been able to get it to work that way yet. So what I've done is start zmprov with 'zmprov 2>&1' which sends stderr through stdout. So now I only have to keep track of one pipe. I still sometimes get stuck and have to figure out a way to get out of it...either by checking for something in the pipe and when I find it exit the loop. Some commands though return nothing if they are successful....so I actually have to send a bogus command in just behind it so I have something to match against. An example is 'mfg' (modifyFolderGrant)....if it succeeds then it just comes back to the prompt and there is nothing in stdout to check for so I just send the word "bogus" in to the prompt...

stdin.puts("bogus")

...and it gives me an error in stdout that I can match against and drop out of the loop.

Anyway...my method of doing it works...though it's a little clunky.

Eventually when I get this all coded I'll maybe drop it into my page on the Zimbra wiki so anyone else interested in doing this with Ruby can not have to go through the headache I have. Maybe find a better way of doing things too...I'm certainly not the worlds greatest Ruby coder.

Matt

[p24t]

I normally don't invoke the interactive zmprov if I'm going to be scripting. I'm not exactly sure if this is slower or not, and I don't really bother to grab stderr or anything. I wouldn't imagine that the interactive mode is intended for scripting.

If I exit from zmprov, then the stream closes, and I can get info from both stdout and stderr. While zmprov interactive is still running, it never closes the stream.

[Chewie71]

It's MUCH faster if you stay in the interactive shell. When you're only doing something for a few accounts it's no big deal, but I'm going to be working with thousands of accounts and it needs to be as fast as possible.

I did a test with a hundred or so accounts and it took well over an hour when I called zmprov every single time I ran a command. It takes a long time for Java(I think it's java) to start every time you open the zmprov shell. I have yet to run a full test on those 100 accounts again with my new method, but it should be drastically faster by staying in interactive mode.

Matt

[Rich Graves]

There will be some learning curve, but I think you'll be better off using a native ruby SOAP library instead of spawning zmprov. See docs/soap.txt (or thereabouts) from the distribution tarball. More work figuring out the APIs, but less work screen-scraping.

[Chewie71]

Rich,

You're probably right....but it means learning SOAP which so far I haven't been able to make much sense of. If you know of any code samples or good tutorials anywhere let me know. Seems that most of the stuff out there about it is pretty vague.

Let me ask this....how does the speed compare to doing it interactively in zmprov?

Matt