Solution to import product data from remotely hosted CSV files into Magento

We use a third party solution to perform this task at present but we would like to expand the functionality with a solution of our own. Here are the basics - Ecommerce platform: Magento Product Data files: CSV accessed via FTP Do all CSV columns have the same name: No How many different files: Right now 20 What type of product data do you wish to import: Product data for the purpose of creating new products and inventory data to update our 'on hand quantity' which is displayed on our live web store How often do you need to update the store: Every 24 hours Ideally, we'd like a solution that performs as follows. We'll call the solution ProductImport (PIMP!) for this case scenario. 1. We create a vendor in Magento 2. We create an attribute set relating to this vendor in Magento 3. We create the vendor in PIMP and confirm the attribute set from Magento to be used with this vendor 4. We specify the location of the vendors CSV files (generally there are two) for both product data and inventory updates and provide the credentials in PIMP 5. PIMP confirms it can access the remote file and returns the columns headers in the CSV file 6. In PIMP, we map the headers in the CSV file to the attributes contained in the attribute set for this vendor 7. PIMP imports the data and pushes it into Magento 8. Same occurs for the vendors inventory file although an attribute set will not need to be assigned as the fields in Magento for inventory quantity and so on are the same for any vendor Considerations - 1. What happens if the product has been updated directly in Magtento. Should we overwrite? For product data - no. For inventory related info - yes. 2. What happens if the product is no longer in the CSV file and still present in Magento? PIMP should delete the product record from Magento. 3. What if PIMP fails to connect to the supplier or Magento? Be good to see an error report or some sort of failure notification. 4. What if you no longer do business with the supplier? They need to be removed from PIMP. We are open to suggestions as to how this should be built - externally hosted solution or an extension in Magento. You MUST have experience with Magento and flat file extraction. Its a pretty common practise but I don't want to be someones guinea pig project =)