Exclude Folders in a File List?

[rcraig]

How would I go about excluding specific folders in a file list? Let me explain what I'm trying to do:

I have a project that does an inventory of our FTP site. It does this by using a simple "file list" action to generate a list of every file on the FTP site. I set it to the root folder of the FTP site and set its "recursive" value to TRUE. This successfully generates a file list containing all files. However, there are some folders I do not need reported. I've tried using a wildcard filter to exclude the folder names, but had no luck, it still reported files in those folders. How could I tell the recursive file list to skip over certain folders?

I know I could set this up a different way, having a separate task to inventory each folder that I want (and simply don't create a task for the folders I don't want). However, due to the number of folders we use, this would be a time-consuming task to implement and update should we ever add/change/delete a folder. I assumed excluding folders would be simple. That'll teach me to assume!

Any help is greatly appreciated. Thanks.

[Support_Mary]

Is this something you need to do right away? Or is it something that can wait? We may be enhancing GoAnywhere Director with an option to include or exclude folders in the future.

We do have a temporary solution that may work for you but it will require more steps to exclude or include changes.

[rcraig]

No, it is not a high priority, but it would be a nice feature to have. Currently I have it set up to File List the entire FTP site, but Date Filter it to only include files that were placed today, thus giving an accurate report of any new files on the FTP site.

Having this new folder filter feature (alliteration anyone?) would allow easy setup of a full FTP inventory as I could just exclude folders I don't want reported, and thus cut down thousands of files in the report.

Right now if I do a full inventory, it reports ALL files, unless I manually exclude certain file names/extensions, which there are hundreds of.

Your temporary solution may be of interest. Could you provide a brief explanation of what would be involved?

Thanks.

[Support_Duane]

It is not exactly simple, but not too difficult. What you need to do is to convert the filelist into a database file that you can read with SQL, allowing you to select only the records that do not have the folder name you specify.

The following xml code will provide a general idea of what you need to do.

Code: Select all

            <createFileList version="1.0" fileListVariable="filesInRoot">
                  <fileset dir="/djohnson" recursive="true" />
            </createFileList>


            <print version="1.0" file="/filelist/filesfound.csv" append="false">
                  <![CDATA[${filesInRoot}]]>
            </print>


            <readCSV version="1.0" inputFile="/filelist/filesfound.csv" outputRowSetVariable="data" skipFirstRow="false" fieldDelimiter="pipe" recordDelimiter="LF" />


            <sql version="1.0" resourceId="Dev61">
                  <query label="create file">
                        <statement>create table qtemp.onefield (fld01 char(256))</statement>
                  </query>
                  <query label="fill file" inputRowSetVariable="${data}">
                        <statement> insert into qtemp.onefield values(?)</statement>
                  </query>
                  <query label="select records" outputVariable="remainingFiles">
                        <statement>select * from qtemp.onefield where fld01 not like &apos;%/${folderName}/%&apos;</statement>
                  </query>
            </sql>


            <writeCSV version="1.0" inputRowSetVariable="${remainingFiles}" outputFile="/filelist/remainingFiles.csv" whenFileExists="overwrite" includeHeadings="false" fieldDelimiter="pipe" recordDelimiter="LF" textQualifier="none" />


For the statement select * from qtemp.onefield where fld01 not like '%/${folderName}/%' you’ll need to either create and fill a variable named folderName or change that value to the actual folder name you wish to exclude.