Hi,
I've been tasked with developing a migration tool that will create files and folders within a DMS. The data to be migrated is in the order of hundreds of gigs (migrated over a number of phases) with potentialy up to 15 to 16 levels of folders in the file system. Essentialy I'll be re-creating the file system within the DMS.
Performance is a key criteria for my customer. With this in mind I have a couple of ideas on how to approach the reading of the file system which I would appreciate some comments on.
1. Use a recursive method to iterate down through the file system untill the lowest level of each branch is reached capturing files and folders as required. Then, recursivly work my way back up the stack to a point where I start down the next branch of the tree.
2. Store the full path to every file throughout the structure in memory and then enumerate the collection of paths, passing the required data to the methods required for the creation of data with the DMS.
Of course, with each of these methods there is an issue of memory as the recursive methods values etc would be stored in memory as would the collection of paths.
So if my logic is correct, method 1 would be a looping type of structure that would work it's way down each branch of the file system untill it reached the lowest level and then work it's way back up the stack to a point at which the process would start over. Method 2 would still need to loop through the file system getting the paths to the files and store those paths in a collection for later access.
I am realy trying to get a feel for the best approach to this type of task. I welcome any suggestions and would especialy appreciate feedback from developers who have been tasked with similar requests.
I've been tasked with developing a migration tool that will create files and folders within a DMS. The data to be migrated is in the order of hundreds of gigs (migrated over a number of phases) with potentialy up to 15 to 16 levels of folders in the file system. Essentialy I'll be re-creating the file system within the DMS.
Performance is a key criteria for my customer. With this in mind I have a couple of ideas on how to approach the reading of the file system which I would appreciate some comments on.
1. Use a recursive method to iterate down through the file system untill the lowest level of each branch is reached capturing files and folders as required. Then, recursivly work my way back up the stack to a point where I start down the next branch of the tree.
2. Store the full path to every file throughout the structure in memory and then enumerate the collection of paths, passing the required data to the methods required for the creation of data with the DMS.
Of course, with each of these methods there is an issue of memory as the recursive methods values etc would be stored in memory as would the collection of paths.
So if my logic is correct, method 1 would be a looping type of structure that would work it's way down each branch of the file system untill it reached the lowest level and then work it's way back up the stack to a point at which the process would start over. Method 2 would still need to loop through the file system getting the paths to the files and store those paths in a collection for later access.
I am realy trying to get a feel for the best approach to this type of task. I welcome any suggestions and would especialy appreciate feedback from developers who have been tasked with similar requests.
